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EXECUTIVE  SUMMARY 


This  report  summarizes  the  Integrated  Icing  Diagnostic  Algorithm  (IIDA)  Assessment  conducted 
at  the  Aviation  Weather  Center  (AWC)  in  Kansas  City,  Missouri,  from  March  through  May  1998. 
Specific  results,  conclusions,  and  recommendations  for  the  assessment  are  detailed  within  this 
report. 

The  Research  Applications  Program  at  the  National  Center  for  Atmospheric  Research 
(NCAR/RAP)  has  developed  the  IIDA  by  combining  a  number  of  icing  detection  techniques  into 
an  integrated  algorithm  that  makes  use  of  the  strengths  of  each  technique  while  simultaneously 
minimizing  individual  weaknesses.  The  IIDA  makes  use  of  satellite  imagery,  numerical  weather 
prediction  model  output,  radar  reflectivity  mosaic,  and  surface  weather  observations.  The 
algorithm  output  consists  of  three-dimensional  grids  of  Icing  Potential  and  Supercooled  Large 
Drop  (SLD)  Potential,  augmented  by  several  derived  intermediate  products  to  aid  in  assessing 
icing  type  and  meteorological  conditions  associated  with  the  icing. 

The  assessment  was  conducted  by  the  Federal  Aviation  Administration  (FAA)  William  J. 

Hughes  Technical  Center,  Weather  Branch  (ACT-320).  The  assessment  took  place  at  AWC 
and  involved  collecting  feedback  from  Area  Forecasters  responsible  for  issuing  aviation 
weather  advisories  and  warnings.  The  purpose  of  the  assessment  was  to  evaluate  the  current 
utility  of  the  integrated  algorithm  and  to  guide  NCAR/RAP  in  making  improvements  to  the 
algorithm.  In  particular,  the  assessment  collected  feedback  from  AWC  forecasters  on  their 
subjective  perception  of  the  value  of  the  current  IIDA  as  well  as  suggestions  for  improvements. 

The  results  of  the  assessment  indicated  that  there  was  value  to  integrating  various  icing 
detection  techniques.  In  addition,  the  IIDA  Icing  Potential  product  appeared  to  provide  the  most 
benefit.  AWC  forecasters  provided  several  suggestions  for  further  enhancements  to  the  IIDA. 
While  the  IIDA  performed  well  when  aircraft  icing  was  the  result  of  very  organized,  synoptic- 
scale  weather  systems,  its  performance  was  not  significantly  different  than  existing  sources  of 
icing  information  available  to  the  AWC  forecasters.  However,  it  was  noted  that  AWC  is  a 
relatively  rich  data  environment,  providing  forecasters  with  a  multitude  of  products  to  choose 
from  in  order  to  assist  in  the  formation  of  aviation  weather  products. 

The  results  of  the  assessment  also  indicated  the  IIDA  SLD  Potential  product  did  not  appear  to 
perform  well.  The  assessment  results  were  not  sufficient  to  provide  reasons  why  the 
performance  may  have  been  poor.  It  is  possible  that  SLD  conditions  did  not  exist  in  the 
atmosphere  during  the  assessment  period  or  the  small  areas  of  SLD  Potential  may  not  be 
represented  within  the  format  of  the  AWC  issued  advisories  and  warnings. 

Recommendations  from  the  assessment  focus  on  further  development  of  the  IIDA  including  the 
development  of  a  forecast  capability  that  incorporates  icing  intensity. 
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1.  INTRODUCTION. 


1.1  BACKGROUND. 

The  Research  Applications  Program  at  the  National  Center  for  Atmospheric  Research 
(NCAR/RAP)  has  conducted  research  and  development  activities  for  improvement  of  in-flight 
icing  detection  and  forecasting  since  1 989.  Efforts  have  focused  upon  applying  algorithms  to 
operational  model  output;  using  Geostationary  Orbiting  Environmental  Satellite  (GOES)  data  for 
detection  of  supercooled  liquid  at  cloud  top;  combining  surface  observations  and  model  outputs 
to  diagnose  supercooled  large  drop  (SLD)  conditions;  and  developing  microphysical 
parameterizations  for  numerical  weather  models.  To  date,  most  of  the  icing  diagnosis  work  has 
utilized  single  instruments  or  models.  By  themselves,  these  individual  techniques  are  not 
adequate  to  diagnose  icing  conditions  in  every  possible  case.  Each  technique  has  its  own 
strengths  and  weaknesses.  A  possible  method  of  providing  an  improved  icing  diagnosis  is  to 
combine  a  number  of  techniques  into  an  integrated  algorithm  that  makes  use  of  the  strengths  of 
each  while  simultaneously  minimizing  individual  weaknesses.  In  recent  years,  NCAR/RAP  has 
used  “fuzzy  logic”  techniques  for  this  sort  of  integration  with  considerable  success.  This 
approach  lends  itself  well  to  icing  problems,  and  is  being  pursued  for  the  development  of  an 
integrated  icing  algorithm. 

For  the  1997-98  winter  season,  NCAR/RAP  prepared  an  Integrated  Icing  Diagnostic  Algorithm 
(I IDA)  which  makes  use  of  GOES-8  satellite  imagery,  model  output  from  the  National  Centers 
for  Environmental  Prediction  (NCEP)  Rapid  Update  Cycle  (RUC),  Next  Generation  Weather 
Radar  (NEXRAD)  reflectivity  mosaic,  and  National  Weather  Service  (NWS)  surface 
observations.  The  algorithm  output  consisted  of  a  three  dimensional  (3-D)  grid  of  icing 
potential,  scaled  from  0  (no  icing)  to  100  (icing  very  likely).  This  was  augmented  by  several 
derived  intermediate  products  to  aid  in  assessing  icing  type  and  the  meteorological  conditions 
associated  with  the  icing.  The  IIDA  currently  does  not  have  a  forecast  capability. 

The  initial  assessment  of  the  IIDA  was  conducted  at  the  Aviation  Weather  Center  (AWC)  from 
March  through  May  1998.  Personnel  from  the  Federal  Aviation  Administration  (FAA)  William  J. 
Hughes  Technical  Center  (FAA  Technical  Center)  Weather  Branch  (ACT-320)  conducted  the 
assessment  in  order  to  evaluate  the  current  utility  of  the  integrated  algorithm  and  to  guide 
NCAR/RAP  in  making  improvements  to  the  algorithm.  In  particular,  the  assessment  collected 
feedback  from  AWC  forecasters  on  their  subjective  perception  of  the  value  of  the  current  IIDA 
as  well  as  suggestions  for  improvements. 

1.2  PURPOSE  OF  REPORT. 

The  purpose  of  this  report  is  to  document  IIDA  assessment  activities,  results,  conclusions,  and 
recommendations.  This  report  will  be  provided  to  NCAR/RAP  to  assist  with  future  development 
of  the  IIDA.  Note  that  NCAR/RAP  is  conducting  IIDA  verification  activities  to  measure  the 
meteorological  accuracy  of  the  product.  NCAR/RAP  will  be  responsible  for  issuing  a  separate 
report  on  their  verification  work. 
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2.  REFERENCE  DOCUMENTS. 


a.  FAA-STD-024B;  US  Department  of  Transportation,  Federal  Aviation  Administration 
Standard;  Content  and  Format  Requirements  for  the  Preparation  of  Test  and  Evaluation 
Documentation. 

b.  Assessment  Plan  for  the  In-flight  Icing  Product  Development  Team  Integrated  Icing 
Diagnosis  Algorithm  (IIDA),  FAA  Technical  Center  and  NCAR,  February  27, 1998. 

3.  PRODUCT  OVERVIEW. 

The  IIDA  is  an  algorithm  designed  to  integrate  various  data  sources  and  provide  a  diagnosis  of 
the  potential  for  in-flight  icing.  Specific  inputs,  outputs,  and  hardware  are  described  in  the 
following  sections. 

3.1  IIDA  COMPONENTS. 

The  IIDA  uses  the  following  data  inputs: 

a.  GOES-8  multi-spectral  satellite  imagery; 

b.  RUC  model  output  (Temperature,  Relative  Humidity,  and  Height); 

c.  NEXRAD  reflectivity  mosaic  (NEXRAD  Information  Dissemination  Service  [NIDS] 
product); 

d.  NWS  surface  observations,  including  cloud  coverage,  ceiling  height,  and  precipitation 
type. 

The  IIDA  output  consists  of: 

a.  3-D  grids  of  Icing  Potential,  scaled  from  0  (no  icing)  to  100  (icing  very  likely), 

b.  3-D  grids  of  SLD  Potential,  scaled  from  0  (no  SLD  icing)  to  100  (SLD  icing  very  likely), 

c.  3-D  grids  of  Icing  Type,  scaled  from  0  (rime)  to  50  (mixed)  to  100  (clear). 

The  IIDA  data  fields  are  scaled  to  match  the  standard  RUC-2  data  grid  (40  kilometer  (km)  in  the 
horizontal  directions  with  40  levels),  with  output  produced  on  an  hourly  basis.  Augmenting  the 
primary  output  grids  are  several  derived  intermediate  products  in  the  form  of  two-dimensional 
(2-D)  gridded  data  fields  intended  to  aid  in  assessing  in-flight  icing.  The  intermediate  2-D 
products  are: 

a.  Cloud  Bases  and  Tops, 

b.  Icing  Bases  and  Tops, 

c.  SLD  Bases  and  Tops. 

For  the  1997-98-winter  season,  the  IIDA  products  were  exclusively  a  diagnosis  of  the  real-time 
conditions;  there  was  no  forecast  capability. 
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3.2  HARDWARE  SYSTEM. 


The  IIDA  software  ran  at  NCAR/RAP  on  a  Sun  Ultra  Sparc  1  workstation.  Output  grids  and 
intermediate  products  were  produced  hourly  and  placed  on  the  NCAR  server.  The  AWC 
obtained  the  output  via  an  automated  file  transfer  process  over  the  Internet.  At  AWC,  the 
output  and  intermediate  products  were  displayed  on  the  Advanced  Weather  Interactive 
Processing  System  for  National  Centers  (N-AWIPS),  AWC’s  meteorological  workstation. 
N-AWIPS  consists  of  a  Hewlett-Packard  workstation  with  dual  monitors  for  the  display  and 
animation  of  meteorological  fields. 

Forecasters  viewed  the  IIDA  in  a  four-panel  N-AWIPS  display.  Three  of  the  panels  were  color- 
filled  contours  of  the  three  IIDA  products  at  each  level  from  1000  millibars  (mb)  to  400  mb.  The 
fourth  panel  showed  Icing  Pilot  Reports  (PIREP).  Also  available  were  single  panel  displays  of 
1000-400  mb  Composite  Maximum  Icing  and  SLD  Potential,  Icing  Bases  and  Tops,  SLD  Bases 
and  Tops,  and  Cloud  Bases  and  Tops. 

4.  ASSESSMENT  DESCRIPTION. 

4.1  ASSESSMENT  SCHEDULE  AND  LOCATION. 

Prior  to  the  formal  start  of  the  assessment,  NCAR  personnel  conducted  training  on  February  1 1 
and  12, 1998.  The  training  included  information  on  in-flight  icing  physics  and  IIDA  background 
information.  However,  due  to  implementation  problems,  the  IIDA  was  not  available  for  hands- 
on  training  by  AWC  personnel.  During  the  same  time  period  as  training,  ACT-320  personnel 
conducted  baseline  observations  and  interviews  with  AWC  area  forecasters  responsible  for 
preparing  Icing  Airmen’s  Meteorological  Statements  (AIRMET)  and  Significant  Meteorological 
Statements  (SIGMET). 

The  actual  IIDA  Assessment  was  divided  into  two  phases,  both  of  which  were  conducted  at 
AWC.  The  first  phase  was  conducted  on  March  27  through  April  10, 1998.  This  phase 
consisted  of  AWC  area  forecasters  completing  a  daily  questionnaire  on  the  performance  of  the 
IIDA  in  regards  to  the  preparation  of  Icing  AIRMETs. 

The  second  phase  of  the  IIDA  assessment  consisted  of  ACT-320  personnel  conducting 
observations  and  interviews  with  AWC  area  forecasters.  These  observations  and  interviews 
were  conducted  on  April  27  and  28, 1998.  Forecasters  completing  a  second  questionnaire, 
separate  from  the  one  described  above  that  assessed  the  overall  utility,  reliability,  and  accuracy 
of  the  IIDA  followed  the  observations  and  interviews.  This  questionnaire  was  administered  from 
May  7  through  15, 1998. 

4.2  PARTICIPANTS. 

Fifteen  AWC  area  forecasters  participated  in  the  first  phase  of  the  IIDA  Assessment  (i.e.,  the 
daily  questionnaire)  and  1 3  forecasters  completed  the  questionnaire  in  the  second  phase.  Of 
the  15  area  forecasters  participating  in  the  assessment,  ACT-320  personnel  were  able  to 
observe  and  interview  10.  The  total  number  of  AWC  area  forecasters  operating  at  AWC  during 
the  assessment  period  was  1 7.  Thus,  the  total  number  of  participants  was  considered  as  a 
representative  sample  of  the  area  forecaster  population. 


3 


4.3  APPROACH. 


The  IIDA  Assessment  consisted  of  obtaining  subjective  feedback  from  AWC  area  forecasters. 
The  forecasters  reviewed  the  IIDA  output  and  intermediate  fields  during  normal  operational 
shifts.  ACT-320  personnel  collected  feedback  on  forecasters’  impressions  of  the  algorithm 
through  the  use  of  questionnaires,  interviews,  and  operational  observations.  Results  were 
determined  from  analyzing  and  summarizing  the  feedback. 

4.3.1  Phase  1. 

The  first  phase  of  the  IIDA  Assessment  addressed  the  day-to-day  performance  of  the  IIDA  and 
its  utility  in  producing  Icing  AIRMETs  and  SIGMETs.  Forecasters  completed  a  daily 
questionnaire  once  per  shift.  The  questionnaire  was  installed  in  an  electronic  format  on  a  PC 
located  in  the  forecaster  work  area  at  AWC.  After  completing  the  questionnaire,  the 
forecaster’s  responses  were  automatically  transferred  to  a  Microsoft  Access  database  on  the 
PC.  ACT-320  personnel  performed  a  daily  file  transfer  to  access  the  most  current  version  of 
the  database. 

4.3.2  Phase  2. 

The  second  phase  of  the  IIDA  Assessment  addressed  the  overall  utility,  reliability,  and  accuracy 
of  the  IIDA  as  perceived  by  AWC  forecasters.  ACT-320  personnel  conducted  observations  and 
interviews,  followed  by  forecaster  completion  of  a  questionnaire  separate  from  the  daily 
questionnaire  used  in  phase  1.  The  observations  and  interviews  were  used  to  solicit 
information  that  may  not  have  been  ratable  within  the  questionnaire  format  used  in  the  first 
phase.  The  questionnaire  addressed  the  overall  utility,  reliability,  and  accuracy  of  the  IIDA. 
Forecasters  completed  the  questionnaire  once  at  the  end  of  the  assessment  period.  The  End- 
of-Assessment  Questionnaire  was  installed  in  an  electronic  format  on  the  same  PC  located  in 
the  forecaster  work  area  at  AWC  that  was  used  for  the  daily  questionnaire.  The  database  was 
transferred  to  ACT-320  personnel  at  the  end  of  the  assessment  period  for  data  analysis. 

4,4  ASSESSMENT  OBJECTIVES. 

The  objectives  of  the  IIDA  assessment  were: 

a.  Determine  if  there  is  value  to  integrating  the  various  intermediate  fields  into  a  single 
output  field; 

b.  Subjectively  assess  the  value  of  the  various  intermediate  products,  specifically: 

1 .  What  components  of  the  IIDA  appear  to  be  providing  the  most  benefit  for  identifying 
areas  of  aircraft  icing? 

2.  What  components  could  be  added  to  improve  algorithm  performance? 

3.  Under  what  situations  does  the  IIDA  perform  well  or  not  perform  well? 

c.  Subjectively  identify  how  the  IIDA  performance  compares  to  current  methods  used  by 
AWC  forecasters  for  identifying  the  presence  of  aircraft  icing  conditions. 


4 


d.  Identify  AWC  data  inputs  and  platforms  so  that  further  development  can  be  tailored  to 
AWC  operations. 

4.5  ASSESSMENT  DESCRIPTION  AND  METHODOLOGY. 

4.5.1  Baseline  Measures. 

Concurrent  with  the  preassessment  training  effort,  baseline  measures  were  obtained  by 
ACT-320  personnel.  The  data  inputs,  platforms,  and  current  procedures  used  by  AWC  area 
forecasters  in  determining  icing  AIRMETs  and  SIGMETs  were  identified.  This  information  was 
collected  in  order  to  provide  a  comparison  between  current  methodologies  and  IIDA.  The 
information  was  obtained  by  ACT-320  personnel  via  forecaster  interviews  and  observations  of 
forecast  operations. 

4.5.2  Daily  Questionnaire. 

The  purpose  of  the  questionnaire  was  to  rate  each  of  the  IIDA  3-D  and  2-D  components,  and 
other  products  that  AWC  forecasters  use  to  produce  Icing  AIRMETs  and  SIGMETs.  Ratings 
were  based  on  a  5-point  Likert  rating  scale.  In  addition,  each  forecaster  recorded  a  subjective 
measure  of  the  Icing  Potential  and  SLD  Potential  as  compared  to  Icing  PIREPs.  Open-ended 
questions  were  also  included  soliciting  suggestions  for  improvements,  note-worthy 
performances,  or  other  comments. 

AWC  forecasters  were  able  to  briefly  note  any  negative  or  positive  performance  aspects  of  the 
algorithm  during  day-to-day  use.  Information  from  the  daily  questionnaire  was  used  to  identify 
specific  instances  of  algorithm  performance  along  with  the  date  and  time  of  the  occurrence.  A 
copy  of  the  questionnaire  is  included  in  appendix  A. 

4.5.3  Interviews. 

The  IIDA  Assessment  included  personal  interviews  of  AWC  area  forecasters  conducted  by 
ACT-320  personnel.  The  intent  of  the  interviews  was  to  solicit  information  that  was  not  ratable 
within  the  questionnaire  format.  Data  from  interview  questions  was  used  to  provide  insight  into 
issues  pertaining  to  the  algorithm  that  were  difficult  or  too  time  consuming  for  a  user  to  write 
down.  In  addition,  more  detailed  information,  clarification  on  reported  problems  or  benefits,  and 
other  pertinent  comments  were  obtained  during  the  interview  process. 

4.5.4  Observations. 

Observations  of  AWC  forecasters’  use  of  IIDA  during  actual  operations  occurred  in  conjunction 
with  the  forecaster  interviews  discussed  in  section  4.5.3.  ACT-320  personnel  used  observation 
logs  to  record  how  and  under  what  circumstances  the  IIDA  was  used,  whether  other  methods 
were  used  to  produce  icing  AIRMETs  and  SIGMETs,  and  any  additional  feedback  from  the 
forecasters  on  duty.  Observations  were  nonobtrusive  to  AWC  operations. 
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4.5.5  End-of-Assessment  Questionnaire. 


In  addition  to  the  observations  and  interviews,  AWC  forecasters  were  provided  the  opportunity 
to  complete  a  final  questionnaire  addressing  the  overall  utility,  reliability,  and  accuracy  of  the 
3-D  output  fields  of  Icing  Potential;  SLD  Potential;  and  Icing  Type.  Users  also  answered 
questions  about  the  utility,  reliability,  and  accuracy  of  the  six  2-D  IIDA  intermediate  product 
fields  consisting  of  Cloud  Bases  and  Tops,  Icing  Bases  and  Tops,  and  SLD  Bases  and  Tops. 

The  questionnaire  also  included  a  number  of  open-ended  questions  soliciting  suggestions  for 
improving  the  algorithm,  including  suggestions  for  possible  additional  inputs  and/or 
combinations  of  inputs.  A  copy  of  the  questionnaire  is  included  in  appendix  B. 

4.5.6  Data  Collection  and  Analysis. 

Data  collection  was  accomplished  by  administering  the  two  separate  questionnaires  previously 
discussed,  supplemented  by  on-site  observations  and  interviews. 

Product  ratings  from  the  daily  questionnaire  were  analyzed  using  frequency  distributions  to 
determine  which  products  (both  of  the  IIDA  and  existing  AWC  capabilities)  were  rated  most 
important  in  the  preparation  of  Icing  AIRMETs  and  SIGMETs.  Since  the  number  of  daily 
questionnaires  completed  by  each  AWC  forecaster  varied,  the  median  was  used  to  determine 
an  overall  individual  user  response  to  each  product  rating.  This  was  done  to  reduce  the  bias 
from  users  who  completed  the  daily  questionnaire  several  times  during  the  assessment  period. 
The  individual  user  response  median  was  used  as  a  single  data  point  in  the  frequency 
distribution  for  the  product  ratings. 

The  median  score  is  the  most  appropriate  measure  of  central  tendency  when  using  ordinal  data 
(e.g.,  questionnaire  ranking  scales)  as  it  relies  on  a  ranking  process.  The  median  is  the  mid¬ 
point  of  the  observations.  When  there  is  an  even  number  of  observations,  no  unique  center 
value  exists,  so  the  mean  of  the  two  middle  observations  is  taken  as  the  median  value. 

The  perceived  accuracy  of  the  Icing  Potential  and  SLD  Potential  from  the  daily  questionnaires 
was  analyzed  using  frequency  distributions  to  determine  the  range  where  the  greatest 
concentration  of  moderate  or  greater  icing  PIREPs  occurred.  Each  individual  response  was 
retained  as  a  separate,  independent,  data  point  since  weather  conditions  changed  from  day  to 
day. 

Ratings  on  product  and  component  utility,  reliability,  and  accuracy  from  the  end-of-assessment 
questionnaire  were  analyzed  using  frequency  distributions.  Each  user  only  completed  the 
questionnaire  once,  thus  there  was  no  need  to  calculate  and  use  the  median  in  these  frequency 
distributions.  Each  response  was  used  as  a  single  data  point  in  the  frequency  distributions. 

In  addition,  comments  to  interview  questions  were  recorded  and  summarized.  Recurring 
comments  were  tabulated. 
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5.  RESULTS  AND  DISCUSSION. 

This  section  presents  the  results  of  the  1 1  DA  Assessment.  Section  5.1  discusses  factors  that 
may  have  affected  the  results.  Section  5.2  discusses  the  results  from  the  Daily  Questionnaire. 
Section  5.3  discusses  results  from  the  Observations  and  Interviews.  Section  5.4  discusses  the 
results  from  the  End-of-Assessment  Questionnaire. 

5.1  FACTORS  AFFECTING  RESULTS. 

5.1.1  Training. 

Due  to  delays  in  implementing  the  IIDA  at  AWC,  NCAR  personnel  did  not  provide  formal 
training  on  the  algorithm  itself.  A  seminar  was  presented  on  the  science  of  in-flight  icing  and  a 
description  of  IIDA.  However,  no  formal  training  (either  in  a  group  setting  or  one-on-one) 
involved  the  actual  IIDA.  As  a  result,  forecaster  familiarity  with  the  IIDA  products  and 
components  appeared  to  be  lacking  in  some  instances.  This  may  have  influenced  forecasters’ 
use  or  non-use  of  the  IIDA  during  the  assessment  period.  Examples  obtained  from  different 
phases  of  the  assessment  supported  forecaster  unfamiliarity.  Specific  examples  included: 

a.  A  forecaster  stating  during  interviews  they  were  not  familiar  with  the  algorithm; 

b.  A  forecaster  did  not  know  Icing  Type  was  available. 

c.  A  forecaster  requested  that  satellite  imagery  be  incorporated  within  the  IIDA. 

5.1.2  Forecaster  Preferences. 

The  AWC  forecasters  operate  in  a  time-crucial  environment,  often  without  significant  time 
available  to  review  new  products.  In  addition,  the  average  AWC  area  forecaster  has  several 
years  experience  in  producing  AIRMETs  and  SIGMETs.  Forecasters  tend  to  use  products  that 
they  have  confidence  in  and  seem  reluctant  to  deviate  from  familiar  products  and  procedures 
once  confidence  has  been  gained.  Thus,  any  product  introduced  to  AWC  forecasters  has  to 
overcome  inherent  biases  resulting  from  the  time-crucial  environment  and  the  use  of  favored 
products.  If  new  products  and  algorithms,  such  as  the  IIDA,  are  introduced  without  verification 
information  demonstrating  potential  improvement  over  existing  products,  forecasters  may  be 
reluctant  to  use  or  even  look  at  a  product. 

5.2  DAILY  QUESTIONNAIRE  RESULTS. 

5.2.1  Product  Rankings. 

As  described  in  section  4.5.2,  forecasters  evaluated  each  of  the  IIDA  3-D  products  and  2-D 
components  as  well  as  current  products  that  AWC  forecasters  use  to  produce  Icing  AIRMETs 
and  SIGMETs.  Ratings  were  based  on  a  5-point  Likert  scale  using  the  following  ratings: 

1  =  The  product  or  component  was  not  used. 

2  =  The  product  or  component  had  no  value. 

3  =  The  product  or  component  had  little  value. 

4  =  The  product  or  component  had  value. 

5  =  The  product  or  component  had  the  highest  value. 
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Results  of  the  forecasters’  responses  were  tabulated  and  are  shown  in  table  1 .  Products  were 
ranked  according  to  the  median  value  of  the  responses  and  the  number  of  users.  As  explained 
in  section  4.5.6,  the  median  of  each  user  was  used  in  the  tabulation  in  order  to  reduce  any  bias 
due  to  some  users  providing  responses  on  more  occasions  than  other  users.  (Note  that  since 
the  median  of  an  individual  user  could  be  a  multiple  of  0.5,  the  median  of  an  individual  product 
could  be  a  multiple  of  0.25.)  Products  with  equal  median  values  were  ranked  according  to  the 
number  of  users  who  actually  used  the  product.  A  product  with  an  equivalent  median  value  as 
another  product  would  be  ranked  higher  (lower)  if  a  greater  (lesser)  number  of  users  used  the 
product. 

The  results  in  table  1  show  that  the  highest  ranked  products  (i.e.,  Graphical  PIREPs,  Textual 
PIREPs,  Rawinsonde  Observations  [RAOB],  and  Satellite  Images)  are  observations,  rather 
than  forecasts.  This  indicates  the  reliance  that  AWC  forecasters  place  on  observations  of 
current  conditions  in  producing  their  AIRMETs  and  SIGMETs.  The  highest  ranking  of  any  IIDA 
product  or  component  was  the  Icing  Potential,  which  ranked  1 1th  out  of  19  products. 

TABLE  1.  RANKINGS  FOR  PRODUCTS  USED  TO  PRODUCE  ICING  AIRMETS  AND 

SIGMETS 


Product 

Rank 

Number  of  Users 

Median  Ranking 

Graphical  PIREPs 

1 

15 

4.00 

Text  PIREPs 

1 

15 

4.00 

RAOB 

1 

15 

4.00 

Satellite 

1 

15 

4.00 

AIRMETs 

5 

13 

4.00 

NCAR/RAP  Icing  Algorithm 

5 

13 

4.00 

Neural  Net  Icing  Product 
(NNICE) 

7 

12 

4.00 

Equivalent  Potential  Vorticity 
(EPV)  Sounding 

8 

7 

4.00 

Constant  Altitude  Plan  Position 
Indicator  (CAPPI) 

9 

13 

3.50 

Model  Temperature  (T)  and 
Relative  Humidity  (RH) 

Soundings 

10 

10 

3.25 

Radar 

11 

15 

3.00 

IIDA  Icing  Potential 

11 

15 

3.00 

IIDA  Icing  Bases  &  Tops 

13 

13 

3.00 

IIDA  SLD  Potential 

14 

12 

3.00 

IIDA  Icing  Type 

15 

11 

3.00 

Stovepipe 

16 

8 

3.00 

IIDA  Cloud  Bases  and  Tops 

17 

12 

2.75 

IIDA  SLD  Bases  and  Tops 

18 

11 

2.50 

Freezing  Precipitation 
Observations  (Precip.  Obs.) 

19 

11 

2.00 

Product  rankings  according  to  forecaster  shifts  are  shown  in  tables  2,  3,  and  4.  The  Day  Shift 
is  from  7:00  am  to  3:00  pm;  the  Evening  Shift  from  3:00  pm  to  1 1 :00  pm;  and  the  Mid-Shift  from 
1 1 :00  pm  to  7:00  am  (all  times  are  Central  Standard  Time). 
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The  results  for  the  Day  Shift  (table  2)  show  the  Icing  Potential  product  rising  to  a  ranking  of  3 
from  1 1  for  all  the  shifts.  The  Evening  Shift  shows  a  slight  increase  from  1 1th  to  9th;  while  the 
Mid-Shift  showed  an  increase  to  7th.  The  reason  for  the  significant  increase  during  the  Day 
Shift  is  currently  not  understood. 

TABLE  2.  DAY  SHIFT  RANKINGS  FOR  PRODUCTS  USED  TO  PRODUCE  ICING  AIRMETS 

AND  SIGMETS 


Product 

Rank 

Number  of  Users 

Median  Ranking 

Satellite 

1 

9 

5.00 

2 

2 

4.50 

IIDA  Icing  Potential 

3 

9 

4.00 

NCAR/RAP  Icing  Algorithm 

4 

8 

4.00 

RAOB 

4 

8 

4.00 

Text  PIREPs 

4 

8 

4.00 

Graphical  PIREPs 

4 

8 

4.00 

AIRMETs 

8 

7 

4.00 

CAPPI 

8 

7 

4.00 

NNICE 

8 

7 

4.00 

Model  T  &  RH  Soundings 

11 

4 

3.50 

Radar 

12 

9 

3.00 

IIDA  Icing  Bases  &  Tops 

13 

8 

3.00 

IIDA  SLD  Potential 

14 

7 

3.00 

IIDA  Icing  Type 

14 

7 

3.00 

IIDA  Cloud  Bases  and  Tops 

16 

6 

2.50 

IIDA  SLD  Bases  and  Tops 

17 

5 

2.00 

Stovepipe 

18 

1 

2.00 

Freezing  Precip.  Obs. 

18 

1 

2.00 

*• 
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TABLE  3.  EVENING  SHIFT  RANKINGS  FOR  PRODUCTS  USED  TO  PRODUCE  ICING 

AIRMETS  AND  SIGMETS. 


Product 

Rank 

Number  of  Users 

Median  Ranking 

Satellite 

1 

9 

4.00 

Graphical  PIREPs 

1 

9 

4.00 

Text  PIREPs 

1 

9 

4.00 

RAOB 

1 

9 

4.00 

AIRMETs 

5 

8 

4.00 

NNICE 

5 

8 

4.00 

II DA  Icing  Bases  &  Tops 

7 

7 

4.00 

EPV  Sounding 

8 

3 

4.00 

II DA  Icing  Potential 

9 

9 

3.50 

NCAR/RAP  Icing  Algorithm 

10 

7 

3.50 

Radar 

11 

9 

3.00 

CAPPI 

12 

7 

3.00 

II DA  Cloud  Bases  and  Tops 

13 

6 

3.00 

II DA  Icing  Type 

14 

5 

3.00 

Model  T&RH  Soundings 

14 

5 

3.00 

Stovepipe 

14 

5 

3.00 

II DA  SLD  Potential 

17 

7 

2.50 

II DA  SLD  Bases  and  Tops 

18 

5 

2.50 

Freezing  Precip.  Obs. 

19 

5 

2.00 

TABLE  4.  MID-SHIFT  RANKINGS  FOR  PRODUCTS  USED  TO  PRODUCE  ICING  AIRMETS 

AND  SIGMETS 


Product 

Rank 

Number  of  Users 

Median  Ranking 

Satellite 

1 

10 

4.00 

Graphical  PIREPs 

2 

9 

4.00 

Text  PIREPs 

2 

9 

4.00 

NCAR/RAP  Icing  Algorithm 

4 

8 

4.00 

RAOB 

4 

8 

4.00 

NNICE 

6 

7 

4.00 

II  DA  Icing  Potential 

7 

10 

3.00 

Radar 

7 

10 

3.00 

II DA  Icing  Bases  &  Tops 

9 

9 

3.00 

CAPPI 

10 

7 

3.00 

AIRMETs 

11 

6 

3.00 

II  DA  Cloud  Bases  and  Tops 

11 

6 

3.00 

Model  T&RH  Soundings 

11 

6 

3.00 

MDA  Icing  Type 

14 

5 

3.00 

15 

3 

3.00 

Stovepipe 

15 

3 

3.00 

IIDA  SLD  Potential 

17 

7 

2.50 

II  DA  SLD  Bases  and  Tops 

18 

6 

2.25 

Freezing  Precip.  Obs. 

19 

5 

2.00 
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The  results  of  IIDA  compared  against  similar  type  products  (i.e.,  model  and  or  observation- 
based  algorithms  which  diagnose  or  forecast  the  existence  of  icing)  are  shown  in  table  5.  The 
highest-ranking  product  is  the  NCAR/RAP  Icing  Algorithm,  which  also  has  the  greatest  longevity 
in  terms  of  availability  for  forecaster  use  of  any  of  the  products  ranked.  The  other  products  also 
appear  to  be  ranked  according  to  the  amount  of  time  they  have  been  in  existence  at  AWC.  The 
only  exception  to  this  is  the  Stovepipe  algorithm.  Comments  during  the  assessment  indicated 
that  forecasters  tend  to  not  perceive  surface  observations  of  freezing  precipitation  (a  major 
component  of  the  Stovepipe  algorithm)  as  a  major  factor  in  determining  AIRMETs  and 
SIGMETs.  This  will  be  discussed  more  in  section  6. 

TABLE  5.  MODEL  AND  OBSERVATION-BASED  ALGORITHM  RANKINGS 


Product 

Rank 

Number  of  Users 

Median  Ranking 

NCAR/RAP 

1 

13 

NNICE 

2 

12 

4.00 

IIDA  Icing  Potential 

3 

15 

3.00 

IIDA  SLD  Potential 

4 

12 

3.00 

Stovepipe 

5 

8 

3.00 

Within  the  IIDA  itself,  the  results  of  the  individual  products  and  components  are  shown  in 
table  6.  The  Icing  Potential,  with  Icing  Bases  and  Tops  ranked  the  highest,  followed  by  SLD 
Potential.  Icing  Type  also  recorded  a  median  value  of  3.0,  but  had  fewer  users  than  the  other 
top-ranking  products  and  components.  The  SLD  Bases  and  Tops  were  ranked  the  lowest, 
possibly  demonstrating  less  forecaster  confidence  in  the  vertical  delineation  of  SLD  conditions. 

TABLE  6.  IIDA  PRODUCTS  AND  COMPONENTS  RANKINGS 


Product 

Rank 

Number  of  Users 

Median  Ranking 

Icing  Potential 

1 

15 

3.00 

Icing  Bases  and  Tops 

2 

13 

3.00 

SLD  Potential 

3 

12 

3.00 

Icing  Type 

4 

11 

3.00 

Cloud  Bases  and  Tops 

5 

12 

2.75 

SLD  Bases  and  Tops 

6 

11 

2.50 

5.2.2  Icing  and  SLD  Potential  Range. 

As  stated  in  section  4.5.2,  forecasters  subjectively  compared  areas  identified  by  the  Icing 
Potential  and  SLD  Potential  products  as  having  icing  with  regions  having  the  greatest 
concentration  of  moderate  or  greater  PIREPs.  Table  7  presents  the  ratings  that  forecasters 
used  in  the  comparison. 
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TABLE  7.  PIREP  COMPARISON  RATING  DEFINITIONS 


Rating 

Definition 

1 

No  PIREPs  were  available 

2 

The  product  did  not  identify  icing  in  regions  where  PIREPs  were  identified 

3 

Product  values  of  0-20  were  where  PIREPs  were  identified 

4 

Product  values  of  20-40  were  where  PIREPs  were  identified 

5 

Product  values  of  40-60  were  where  PIREPs  were  identified 

6 

Product  values  of  60-80  were  where  PIREPs  were  identified 

7 

Product  values  of  80-100  were  where  PIREPs  were  identified 

The  results  of  the  Icing  Potential  comparison  are  shown  in  figure  1 .  Frequency  distributions  are 
used  to  identify  where  the  greatest  concentration  of  moderate  or  greater  icing  PIREPs  occurred 
in  comparison  to  the  Icing  Potential  range.  Figure  la  includes  the  cases  when  PIREPs  were 
not  available  to  the  AWC  area  forecasters  (value  1 .0  on  the  horizontal  axis  in  figure  la).  Many 
of  these  cases  may  have  occurred  during  the  late  night  when  air  traffic  is  at  a  minimum.  Figure 
1b  removes  the  cases  when  PIREPs  were  not  available  and  only  shows  the  results  when 
PIREPs  were  available.  This  latter  result  shows  that  51  percent  of  the  cases  had  the  greatest 
concentration  of  moderate  or  greater  PIREPs  corresponding  to  Icing  Potential  ranges  of  greater 
than  60. 


(a) 


(b) 


Icing  Potential  Range  Performance  Icing  Potential  Range  Performance 

n=52  n=35 


Icing  Potential  Range  Icing  Potential  Range 


FIGURE  1.  FREQUENCY  DISTRIBUTION  OF  THE  COMPARISON  BETWEEN  ICING 
POTENTIAL  AND  MODERATE  OR  GREATER  PIREPS 


Note:  (a)  all  cases  (number  of  responses,  n  =  52)  and  (b)  cases  with  no  PIREPs  available 
removed  (n  =  35).  Horizontal  axis  values  are  defined  in  table  7. 

Results  for  the  SLD  Potential  comparison  with  moderate  or  greater  PIREPs  are  shown  in 
figure  2.  The  range  values  are  the  same  as  given  in  table  7.  Figure  2a  includes  the  cases 
when  PIREPs  were  not  available.  Figure  2b  removes  the  cases  when  PIREPs  were  not 
available.  The  results  demonstrate  that  60  percent  of  the  time,  the  SLD  Potential  did  not 
identify  the  greatest  concentration  of  moderate  or  greater  PIREPs.  However,  it  is  not  known 
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whether  this  result  is  from  the  performance  of  the  product  or  due  to  a  lack  of  actual  SLD 
conditions  in  the  atmosphere  during  the  assessment  phase.  Section  6  addresses  this  situation 
further. 

(a)  (b) 

SLD  Potential  Range  Performance  SLD  Potential  Range  Performance 


n=52  n=35 


1.0  2.0  3.0  4.0  5.0  6.0  7.0  2.0  3.0  4.0  5.0  6.0  7.0 


SLD  Potential  Range  SLD  Potential  Range 

FIGURE  2.  FREQUENCY  DISTRIBUTION  OF  THE  COMPARISON  BETWEEN  SLD 
POTENTIAL  AND  MODERATE  OR  GREATER  PIREPS 

Note:  (a)  all  cases  and  (b)  cases  with  no  PIREPs  available  removed.  Horizontal  axis  values 
are  defined  in  table  7. 

5.2.3  IIDA  Added  Value. 

AWC  area  forecasters  were  asked  to  comment  on  whether  the  IIDA  provided  additional 
knowledge  to  icing  situations  that  could  not  have  been  perceived  from  existing  products.  Out  of 
a  possible  52  responses,  forecasters  provided  comments  on  41  occasions  (a  79-percent 
response  rate).  Overall  responses  were  negative,  with  the  majority  indicating  that  the  IIDA  did 
not  provide  anything  beyond  what  other  diagnostics  and  algorithms  provided.  In  addition, 
forecasters  stated  that  on  several  occasions  IIDA  overforecasted  (as  do  many  other  current 
techniques);  on  occasion  did  not  identify  isolated  moderate  reports  of  icing;  and  depicted  the 
cloud  tops  too  high.  Other  single  responses  included  the  icing  tops  were  too  low;  IIDA  depicted 
icing  in  cirrus  clouds;  and  bases  were  too  low. 

The  few  positive  responses  indicated  that  on  two  occasions  during  the  2-week  daily 
questionnaire  period,  IIDA  identified  low-level  icing  situations  whereas  other  techniques  did  not. 
On  two  other  occasions,  the  IIDA  identified  icing  areas  better  than  the  NCAR/RAP  algorithm. 

Unedited  forecaster  comments  on  whether  the  IIDA  provided  additional  knowledge  to  icing 
situations  are  listed  in  their  entirety  in  appendix  C. 

5.3  OBSERVATIONS  AND  INTERVIEW  RESULTS. 

After  the  daily  questionnaire  phase  had  been  completed,  ACT-320  personnel  observed  and 
interviewed  10  of  the  17  AWC  area  forecasters.  Structured  interview  questions  were  used  to 
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allow  forecasters  to  expand  upon  or  clarify  information  obtained  from  the  daily  questionnaire. 
The  interview  questions  and  summarized  responses  are  provided  in  the  following  sections. 
Complete  forecaster  responses  to  the  structured  interview  questions  are  included  in 
appendix  D. 

5.3.1  Question  #1. 

Did  you  use  the  1 1  DA  for  producing  Icing  AIRMETs/SIGMETs?  Please  explain  why  or  why  not. 

Five  respondents  commented  that  they  did  not  use  IIDA  very  much.  The  reasons  for  not  using 
IIDA  included: 

a.  There  was  no  forecast  as  part  of  IIDA; 

b.  Existing  AWC  products  (such  as  PIREPs  and  surface  observations)  were  sufficient; 

c.  A  forecaster  was  not  familiar  with  the  algorithm; 

d.  New  products  are  not  used  unless  they  immediately  demonstrate  superior  results; 

e.  Availability  problems; 

f.  A  new  area  forecaster  was  becoming  familiar  with  AWC  systems; 

g.  One  forecaster  did  not  predominantly  work  forecast  shifts  during  the  assessment. 

Five  respondents  commented  that  they  did  use  the  IIDA  for  producing  Icing  AIRMETs. 
Explanations  included: 

a.  It  was  useful  as  a  starting  point  in  determining  where  current  icing  existed; 

b.  It  was  useful  during  midnight  shifts  when  PIREPs  were  at  a  minimum. 

It  should  be  noted  that  forecasters  who  used  the  IIDA  in  producing  Icing  AIRMETs  indicated 
that  the  IIDA  output  was  very  similar  to  existing  AWC  products  and  that  a  forecast  component 
would  have  been  more  beneficial  in  producing  an  AIRMET. 

No  area  forecasters  reported  that  they  used  IIDA  to  produce  Icing  SIGMETs.  SIGMETs  tend  to 
be  driven  by  PIREPs  of  severe  icing  and  forecasters  rarely  issue  SIGMETs  based  only  upon 
guidance  information. 

5.3.2  Question  #2. 

How  did  IIDA  information  compare  to  other  icing  information  sources? 

Overall,  7  out  of  10  AWC  area  forecasters  interviewed  responded  negatively.  Many  of  the 
responses  indicated  that  the  IIDA  was  too  similar  to  existing  sources  and  did  not  provide  any 
information  that  was  not  already  available.  Specific  comments  included: 

a.  IIDA  was  too  similar  to  existing  products  with  no  new  information  presented; 

b.  IIDA  did  not  reduce  the  area  coverage  of  icing  potential  --  large  areas  were  covered  like 
other  algorithms  do; 

c.  IIDA  improved  only  a  little  upon  the  NCAR/RAP  algorithm; 
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d.  A  detection  product  does  not  help  --  a  forecast  is  needed; 

e.  Meso  Eta  output  is  preferred  since  the  RUC  (which  is  used  in  the  1 1  DA)  tends  to  not  do 
as  well  in  depicting  moisture. 

Three  respondents  compared  1 1  DA  favorably  to  other  icing  information  sources.  Comments 
included: 

a.  The  1 1  DA  is  a  good  diagnostic  tool  with  most  of  the  icing  appearing  to  occur  in  the  Icing 
Potential  range  of  70  and  higher; 

b.  1 1  DA  appeared  to  reduce  the  coverage  given  by  the  Stovepipe  algorithm; 

c.  The  IIDA  was  not  compared  to  other  sources,  however,  the  integration  of  information 
was  useful  and  would  be  helpful  during  data  sparse  times.  Normally,  the  forecaster  must  look 
at  several  profiles  of  icing  information. 

5.3.3  Question  #3. 

What  components,  if  any,  did  you  find  most  useful?  Why? 

Overall,  AWC  forecasters  found  the  Icing  Potential  to  be  the  most  useful  product,  followed  by 
Icing  Bases  and  Tops,  and  Cloud  Bases  and  Tops.  Specific  comments  included: 

a.  Six  respondents  reported  that  Icing  Potential  was  most  useful  since  it  appeared  to  be 
more  accurate  than  other  components  (e.g.,  SLD  Potential); 

b.  Four  respondents  reported  that  Icing  Bases  and  Tops  were  useful; 

c.  Two  respondents  reported  that  Cloud  Bases  and  Tops  were  useful  since  they  appeared 
to  be  accurate. 

Although  the  question  asked  for  components  that  were  useful,  many  respondents  commented 
on  components  they  found  least  useful,  inadequate,  or  unreliable.  These  included: 

a.  Six  respondents  reported  that  SLD  components  (potential,  tops,  and  bases)  were  not 
useful  or  needed; 

b.  Three  respondents  stated  that  Icing  Type  was  not  useful  or  very  applicable; 

c.  One  respondent  stated  IIDA  was  not  useful  because  it  lacked  a  forecast  capability; 

d.  One  respondent  stated  existing  algorithms  identify  the  potential  for  icing  --  an  algorithm 
for  icing  intensity  is  needed; 

e.  One  respondent  stated  that  Icing  Tops  was  less  useful  than  expected  since  icing  and 
cloud  tops  were  4,000’  -  6,000’  too  high; 

f.  One  respondent  stated  that  SLD  bases  and  tops  were  not  useful  due  to  the  lack  of 
PIREPs  to  substantiate  them. 
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5.3.4  Question  #4. 


Describe  typical  situations,  if  any,  where  the  algorithm  and/or  its  components  performed  well. 

Overall,  the  IIDA  appeared  to  perform  best  when  icing  was  the  result  of  organized,  synoptic- 
scale  weather  systems.  Specifically: 

a.  Four  respondents  reported  that  IIDA  performed  well  with  well  defined  or  organized, 
synoptic  situations.  However,  it  was  noted  that  other  icing  information  sources  also  performed 
well  under  these  situations; 

b.  One  respondent  reported  that  although  IIDA  performed  well  with  organized  synoptic 
situations,  the  icing  situation  was  already  known  due  to  pattern  recognition  built  by  experience; 

c.  Six  respondents  did  not  identify  any  situations  where  the  IIDA  performed  well. 

5.3.5  Question  #5 


Describe  typical  situations,  if  any,  where  the  algorithm  and/or  its  components  performed  poorly. 

Forecasters  identified  a  variety  of  situations  where  the  IIDA  appeared  to  perform  poorly. 

Several  of  these  situations  involved  cloud  levels.  The  SLD  Potential  product  and  components 
were  also  identified  as  performing  poorly.  Specific  responses  included: 

a.  IIDA  did  not  always  work  well  identifying  low  level  icing  (i.e.,  less  than  10,000’)  --  the 
lower  the  icing  the  worse  the  performance; 

b.  IIDA  would  often  identify  icing  when  low-topped  clouds  existed  1 ,000  -  2,000’  below  the 
freezing  level  (i.e.,  in  warm  temperatures); 

c.  No  icing  was  identified  above  1 6,000’  in  the  vicinity  of  the  Rockies; 

d.  IIDA  did  not  identify  any  icing  with  a  very  apparent  icing  producing  synoptic  system  with 
layered  clouds  in  the  Pacific  Northwest; 

e.  West  area  forecasters  noted  that  terrain  influences  many  icing  situations  -  IIDA,  and  in 
particular  SLD  Potential,  did  not  seem  to  identify  icing  in  the  west  region; 

f.  One  respondent  reported  underdetection  of  icing  at  low  levels  in  the  northern  Rockies; 

g.  One  respondent  reported  that  icing  bases  and  tops  appeared  to  have  a  low  bias  in  the 
west; 

h.  One  respondent  reported  that  cloud  tops  appeared  to  be  lower  than  their  actual  heights, 
while  cloud  bases  appeared  accurate. 
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Specific  SLD-related  responses  included: 

a.  One  respondent  noted  that  the  SLD  Potential  and  Icing  Type  did  not  appear  to  be 
correlated; 

b.  One  respondent  reported  that  SLD  Potential  greatly  reduced  the  identified  icing  areas  by 
producing  “islands”  of  icing,  but  the  product  does  not  appear  to  relate  to  anything; 

c.  One  respondent  reported  that  SLD  areas  were  too  small  to  meet  AIRMET  criteria. 


5.3.6  Miscellaneous  Responses. 

Many  of  the  AWC  area  forecasters  offered  comments  in  addition  to  the  structured  interview 
questions.  Many  of  the  comments  concerned  product  improvements  and  are  summarized  as 
follows: 


a.  Seven  of  the  ten  respondents  identified  a  forecast  out  to  12-1 8  hours  as  a  necessary 
component; 

b.  Three  respondents  reported  reliability  issues  as  having  a  negative  impact  citing  long 
down  times,  software  glitches,  and  product  update  problems; 

c.  Two  respondents  recommended  that  vertical  velocity  fields  be  added; 

d.  Two  respondents  reported  that  IIDA  display  products  needed  smoothing  to  remove  the 
blocky  appearance; 

e.  Two  respondents  reported  that  testing  should  be  done  during  the  middle  of  the  winter 
season  in  order  to  obtain  a  fair  assessment; 

f.  One  respondent  recommended  removing  convective  areas  since  convective-related 
icing  is  not  an  AIRMET  criterion; 

g.  One  respondent  identified  icing  intensity  as  a  necessary  improvement; 

h.  One  respondent  suggested  thresholding  in  order  to  decrease  the  identified  areas; 

i.  One  respondent  reported  that  a  better  way  of  depicting  multiple  cloud  layers  is  needed; 

j.  One  respondent  reported  that  the  questionnaire  needs  to  be  located  at  the  forecaster’s 
work  area. 
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5.4  END-OF-ASSESSMENT  QUESTIONNAIRE. 


Results  for  the  End-of-Assessment  Questionnaire  are  given  in  the  following  sections.  As  stated 
in  section  4.5.5,  products  and  components  were  rated  in  the  areas  of  Utility,  Reliability,  and 
Accuracy.  Results  are  shown  for  each  IIDA  product  and  component  that  was  available  to  AWC 
area  forecasters.  Definitions  used  in  the  rating  are  in  table  8,  while  the  rating  scale  used  for  all 
three  areas  is  in  table  9.  Note  that  in  many  cases  a  middle  rating  of  “3  =  Borderline”  is 
construed  as  neutral,  however,  for  the  IIDA  Assessment  the  rating  is  slightly  positive  due  to  the 
use  of  adequate  in  the  definition. 

TABLE  8.  QUESTIONNAIRE  RATING  DEFINITIONS 


Rating  Area  Definitions 

Utility  -  The  algorithm  or  component  is  useful,  meets  job  requirements  and  responsibilities  and 
aids  in  detecting  and  diagnosing  areas  of  in-flight  icing.  An  algorithm  or  component  may  be 
useful,  but  not  accurate  in  detecting  or  diagnosing  areas  of  in-flight  icing. 

Reliability  -  Algorithm/components  are  consistently  working  without  problems.  Data  is 
provided  consistently. 

Accuracy  -  This  refers  to  perceived  accuracy  of  the  algorithm  or  component.  An  algorithm  or 
component  may  be  perceived  as  accurate,  but  not  useful  in  detecting  or  diagnosing  areas  of  in¬ 
flight  icing. 

TABLE  9.  QUESTIONNAIRE  RATING  SCALE  DEFINITIONS 

Rating  Scale  Definitions 

1  =  Largely  Unacceptable  -  This  response  indicates  the  algorithm/component  being  assessed 
consistently  impedes  your  ability  to  meet  the  requirements  of  your  job;  likely  to  lead  to 
degradation  of  job  performance. 

2  =  Barely  Unacceptable  -  This  response  indicates  the  algorithm/component  being  assessed 
frequently  impedes  your  ability  to  meet  the  requirements  of  your  job;  may  lead  to  degradation  of 
job  performance. 

3  =  Borderline  -  This  response  indicates  that,  although  the  algorithm/component  being 
assessed  is  adequate,  minor  improvements  would  make  it  more  helpful. 

4  =  Barely  Acceptable  -  This  response  indicates  the  algorithm/component  being  assessed 
frequently  enhances  your  ability  to  meet  the  requirements  of  your  job;  may  lead  to  enhanced 
job  performance. 

5  =  Largely  Acceptable  -  This  response  indicates  the  algorithm/component  being  assessed 
consistently  enhances  your  ability  to  meet  the  requirements  of  your  job;  likely  to  lead  to 
enhanced  job  performance. 

NA  -  you  have  never  used  the  algorithm/component  in  question. 

5.4.1  Icing  Potential. 

Results  for  the  Icing  Potential  product  are  shown  in  figure  3.  Utility  ratings,  shown  in  figure  3a, 
indicate  that  the  AWC  forecasters  perceived  the  product  as  useful  with  75  percent  of 
respondents  giving  a  positive  rating  (i.e.,  3  or  higher).  Reliability  ratings,  shown  in  figure  3b, 
indicate  that  66  percent  of  the  respondents  gave  the  product  a  positive  rating.  However,  25 
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percent  of  the  respondents  rated  the  reliability  as  impeding  their  ability  to  meet  the 
requirements  of  their  job  or  degrading  their  job  performance.  It  should  be  noted  that  IIDA  was 
not  provided  from  a  24-hour  per  day  operation,  thus,  around-the-clock  availability  for  the 
assessment  was  not  to  be  expected.  Accuracy  ratings,  shown  in  figure  3c,  indicate  that  the 
majority  of  AWC  forecasters  (91  percent)  perceived  the  Icing  Potential  as  being  accurate. 
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FIGURE  3.  ICING  POTENTIAL  QUESTIONNAIRE  RESULTS 

Note:  In  each  figure,  the  distribution  of  results  is  shown,  the  total  number  (n)  of  responses,  and 
the  percentage  of  responses  for  each  rating  scale. 
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5.4.2  SLD  Potential. 


Results  for  the  SLD  Potential  product  are  shown  in  figure  4.  In  contrast  to  the  mostly  favorable 
ratings  given  to  the  Icing  Potential,  the  preponderance  of  ratings  for  the  SLD  Potential  were 
negative  in  each  of  the  three  rating  areas.  Utility  ratings,  shown  in  figure  4a,  indicate  a  mixed 
reaction  to  the  usefulness  of  the  SLD  Potential.  However,  the  majority  of  the  responses  were  3 
(i.e.,  borderline)  or  below,  indicating  that  AWC  forecasters  believe  improvements  would  be 
helpful.  Reliability  ratings,  shown  in  figure  4b,  indicate  that  the  majority  of  forecasters  believed 
the  reliability  to  be  borderline  or  below.  Accuracy  ratings,  shown  in  figure  4c,  indicate  that  over 
66  percent  of  forecasters  rated  the  accuracy  of  the  SLD  Potential  product  as  unacceptable 
whereas  the  remaining  33  percent  rated  the  product  as  “borderline.”  See  figure  3  for  a 
description  of  the  figures. 
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FIGURE  4.  SLD  POTENTIAL  QUESTIONNAIRE  RESULTS 
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Results  for  the  Icing  Type  product  are  shown  in  figure  5.  Overall,  the  Icing  Type  product 
received  negative  ratings  in  two  of  the  three  areas.  Utility  ratings,  shown  in  figure  5a,  indicate 
that  the  majority  of  the  respondents  (55  percent)  felt  the  product  impeded  their  ability  to  meet 
their  job  requirements  (i.e.,  ratings  of  1  and  2).  Reliability  ratings,  shown  in  figure  5b,  indicate 
that  the  majority  of  the  AWC  area  forecasters  rated  the  reliability  as  being  adequate.  Accuracy 
ratings,  shown  in  figure  5c,  indicate  that  77  percent  of  respondents  perceived  the  product  as 
not  accurate  (i.e.,  ratings  of  1  and  2).  See  figure  3  for  a  description  of  the  figures. 
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FIGURE  5.  ICING  TYPE  QUESTIONNAIRE  RESULTS 
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The  Icing  Bases  and  Tops  were  rated  separately  from  each  other.  Results  for  the  Icing  Bases 
component  are  shown  in  figure  6.  Utility  ratings,  shown  in  figure  6a,  were  equally  distributed 
around  the  Borderline  rating.  Forecaster  comments  indicated  that  many  rely  on  other  sources 
to  determine  the  icing  base  (most  notably,  freezing  levels  from  observed  soundings).  Reliability 
ratings,  shown  in  figure  6b,  indicate  a  negative  tendency  in  the  ratings  where  only  one 
forecaster  rated  Icing  Base  as  clearly  positive.  Accuracy  ratings,  shown  in  figure  6c,  also 
indicate  an  overall  negative  rating.  For  both  reliability  and  accuracy,  most  ratings  fell  within  the 
borderline  range.  See  figure  3  for  a  description  of  the  figures. 

(a)  (b) 


Icing  Base  Utility  Rating  Icing  Base  Reliability  Rating 

n=11  n=11 


Rating 

FIGURE  6.  ICING  BASES  QUESTIONNAIRE  RESULTS 
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In  comparison  to  the  Icing  Bases,  the  utility  of  the  Icing  Tops,  shown  in  figure  7a,  exhibits  an 
overall  positive  rating.  The  difference  in  ratings  for  Icing  Bases  and  Tops  may  be  due  to 
forecasters’  confidence  in  existing  observation-based  sources  of  icing  bases  (such  as 
soundings  with  the  freezing  level).  However,  a  direct  observation  of  Icing  Tops  is  not  available, 
thus,  forecasters  do  not  have  a  similar  product  for  tops  information.  This  may  account  for  the 
difference  in  the  utility  ratings.  Reliability  ratings,  shown  in  figure  7b,  demonstrate  an  adequate 
rating.  Accuracy  ratings,  shown  in  figure  7c,  indicate  a  slightly  negative  rating,  where  67 
percent  of  forecasters  rated  the  Icing  Tops  borderline  or  below.  See  figure  3  for  a  description 
of  the  figures. 
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FIGURE  7.  ICING  TOPS  QUESTIONNAIRE  RESULTS 
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SLD  Bases  and  Tops  were  rated  separately  from  each  other.  The  number  of  respondents  for 
the  SLD  Bases  and  Tops  was  the  lowest  of  the  1 1  DA  products  and  components.  The  low 
number  of  respondents  may  have  been  due  to  unfamiliarity  (see  section  5.1.1).  Ratings  for 
SLD  Bases  are  shown  in  figure  8.  Utility  ratings,  shown  in  figure  8a,  indicate  a  bimodal 
response;  either  forecasters  generally  thought  the  product  frequently  enhanced  their  ability  or  it 
consistently  impeded  their  ability.  Reliability  ratings,  shown  in  figure  8b,  indicate  a  negative 
tendency  in  the  ratings  where  50  percent  of  respondents  rated  the  reliability  as  unacceptable 
and  37  percent  as  borderline.  Accuracy  ratings,  shown  in  figure  8c,  also  indicate  a 
predominantly  negative  rating,  although  two  respondents  (25  percent)  found  the  accuracy 
acceptable.  See  figure  3  for  a  description  of  the  figures. 
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FIGURE  8.  SLD  BASES  QUESTIONNAIRE  RESULTS 
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Overall  ratings  for  the  SLD  Tops,  shown  in  figure  9,  tended  to  be  negative.  Utility  ratings  for  the 
SLD  Tops,  shown  in  figure  9a,  do  not  exhibit  the  same  bimodal  distribution  as  the  SLD  Bases. 
Instead,  the  responses  indicate  an  overall  negative  rating  for  the  usefulness  of  the  product, 
where  over  57  percent  rated  utility  negatively.  Reliability  ratings,  shown  in  figure  9b,  also 
indicate  a  general  negative  rating.  Accuracy  ratings,  shown  in  figure  9c,  indicate  a  significant 
negative  tendency,  with  only  one  positive  response.  Again,  note  the  small  number  of 
respondents  for  SLD  Tops.  See  figure  3  for  a  description  of  the  figures. 
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FIGURE  9.  SLD  TOPS  QUESTIONNAIRE  RESULTS 
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Cloud  Bases  and  Tops  were  rated  separately  from  each  other.  Ratings  for  the  Cloud  Bases  are 
shown  in  figure  10.  Utility  ratings,  shown  in  figure  10a,  indicate  a  wide  distribution  of  ratings 
with  no  preferred  response.  Reliability  ratings,  shown  in  figure  10b,  have  an  equivalent 
distribution  with  no  preferred  response.  Accuracy  ratings,  shown  in  figure  10c,  indicate  that  the 
majority  of  respondents  were  satisfied  with  the  accuracy  of  the  product;  however,  minor 
improvements  would  enhance  the  product.  See  figure  3  for  a  description  of  the  figures. 
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FIGURE  10.  CLOUD  BASES  QUESTIONNAIRE  RESULTS 


26 


The  ratings  for  Cloud  Tops  are  shown  in  figure  11.  In  contrast  to  the  Cloud  Bases,  the 
responses  to  the  Cloud  Tops  exhibit  a  definite  overall  positive  response.  Utility  ratings,  shown 
in  figure  11a,  demonstrate  an  overall  positive  response  with  seven  of  the  nine  respondents 
rating  the  utility  of  the  Cloud  Tops  as  adequate  or  higher.  Reliability  ratings,  shown  in  figure 
11b,  indicate  a  tendency  for  respondents  to  rate  the  reliability  as  adequate  or  higher.  Accuracy 
ratings,  shown  in  figure  11c,  also  demonstrate  an  overall  77  percent  positive  response  rate, 
with  55  percent  of  the  respondents  rating  the  product  as  frequently  or  consistently  enhancing 
their  ability  to  meet  the  requirements  of  their  job  (i.e.,  ratings  of  4  and  5).  See  figure  3  for  a 
description  of  the  figures. 
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FIGURE  1 1 .  CLOUD  TOPS  QUESTIONNAIRE  RESULTS 
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5.4.7  Product  and  Component  Improvements. 


In  addition  to  product  and  component  ratings,  forecasters  provided  comments  on  potential 
improvements  to  the  IIDA  and  its  components.  Complete  comments  are  listed  in  appendix  E. 
Potential  improvements  included: 

a.  Add  icing  intensity  as  an  output; 

b.  Add  vertical  motion  as  input; 

c.  Add  atmospheric  stability  as  input; 

d.  Include  models  other  than  the  RUC  that  may  have  improved  relative  humidity  fields; 

e.  Remove  convectively  related  icing  reports  from  the  display; 

f.  Include  a  forecast  component  out  to  12  hours; 

g.  Smooth  the  output  to  remove  the  “blockiness”  of  the  products; 

h.  Improve  the  reliability  and  availability. 

In  addition,  respondents  noted  that: 

a.  Icing  Tops  in  the  central  US  are  consistently  too  high  by  4,000  -  6,000  feet  --  Icing  Tops 
and  Cloud  Tops  could  be  very  useful  if  their  accuracy  is  improved; 

b.  IIDA  seemed  to  have  a  hard  time  in  low  cloud  top  situations  --  could  be  a  result  of  RUC 
relative  humidity  problems; 

c.  It  would  be  useful  to  include  a  symbol  at  certain  grid  points,  which  has  the  base  and  top, 
depicted  and  that  could  be  overlaid  on  the  Icing  Potential  or  other  products. 

5.4.8  Added  Value  of  IIDA. 

As  part  of  the  End-of-Assessment  Questionnaire,  forecasters  were  asked  to  determine  if  the 
IIDA  provided  any  added  value  above  existing  products.  Specific  questions  and  summarized 
comments  are  provided  below.  Complete  responses  are  given  in  appendix  F. 

Question:  Does  the  integrated  product  provide  any  value  beyond  having  the  individual 
components  by  themselves?  Please  explain.  Provide  examples  if  necessary. 

Nine  respondents  answered  yes  (i.e.,  integration  provides  value  over  individual 
components),  while  four  respondents  answered  no  (over  a  2  to  1  margin).  Positive 
responses  indicated  that  integration  is  capable  of  reducing  the  time  in  determining  an 
Icing  AIRMET.  Forecasters  noted  that  there  is  often  little  time  to  devote  to  forecasts,  so 
integration  is  helpful  as  long  as  the  components  are  accurate.  Negative  responses 
indicated  that  the  integrated  product  was  too  similar  to  existing  products. 

Question:  Does  the  IIDA  provide  value  above  what  you  currently  use  to  create  Icing  AIRMETs 
(e.g.,  satellite,  radar,  model  soundings,  PIREPs)?  How? 

Most  respondents  answered  that  the  IIDA  did  not  provide  value  above  current  methods 
(12  no  responses,  1  yes  response).  Responses  indicated  that  IIDA  output  was  too 
similar  to  existing  products  (e.g.,  icing  areas  were  large),  thus  it  did  not  provide 
additional  value.  In  addition,  due  to  the  lack  of  an  IIDA  forecast  component,  existing 
products  were  reported  to  be  more  useful. 
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Question:  Does  the  IIDA  reduce  the  amount  of  time  necessary  to  create  an  Icing  AIRMET? 
How? 


Most  respondents  answered  that  IIDA  did  not  reduce  the  time  to  create  an  Icing 
AIRMET  (11  no  responses,  2  yes  responses).  Criticism  included  the  lack  of  a  forecast 
component,  the  lack  of  an  intensity  measure,  and  problems  with  reliability.  Even  though 
respondents  answered  negatively,  forecasters  did  note  that  if  confidence  in  the  output 
was  increased  (through  longer  use  or  improved  accuracy),  there  was  potential  for 
timesavings. 

6.  CONCLUSIONS. 

6.1  ASSESSMENT  OBJECTIVES. 

Section  4.4  identified  the  objectives  for  the  Integrated  Icing  Diagnostic  Algorithm  (IIDA) 
Assessment.  Individual  objectives  are  restated  in  the  following  sections  along  with  conclusions 
addressing  each  specific  objective. 

Objective:  Determine  if  there  is  value  to  integrating  the  various  intermediate  fields  into  a  single 
output  field. 

The  assessment  results  indicated  that  there  was  value  to  integrating  fields.  Integration  was 
capable  of  reducing  the  amount  of  time  needed  to  prepare  icing  Airmen’s  Meteorological 
Statements  (AIRMETs)  and  Significant  Meteorological  Statements  (SIGMETs).  However,  in 
order  for  the  IIDA  to  accomplish  the  added  value,  enhancements  such  as  the  inclusion  of 
forecast  and  intensity  fields,  need  to  be  included.  In  addition,  Aviation  Weather  Center  (AWC) 
area  forecasters  need  to  have  confidence  in  the  integrated  output,  either  through  verification 
results  demonstrating  superior  performance  or  long-term  use. 

Objective:  What  components  of  the  IIDA  appear  to  be  providing  the  most  benefit  for  identifying 
areas  of  aircraft  icing? 

The  Icing  Potential  Product  appeared  to  provide  the  most  benefit.  AWC  area  forecasters 
ranked  the  product  higher  than  other  IIDA  products  and  components.  Interviews  and 
observations  confirmed  that  the  Icing  Potential  was  the  most  useful.  The  product  was 
especially  useful  when  Pilot  Reports  (PIREPs)  were  not  available. 

The  Icing  Tops  and  Cloud  Tops  were  also  rated  useful  as  determined  from  the  end-of- 
assessment  questionnaire.  Why  tops  were  rated  useful  and  not  bases  may  be  related  to  the 
use  of  existing  information,  such  as  surface  observations,  soundings,  and  measured  freezing 
levels,  to  identify  icing  and  cloud  bases.  Forecasters  are  reasonably  confident  of  these 
observations.  Tops,  however,  are  not  included  as  part  of  surface  observations  and  there  may 
be  less  confidence  in  existing  capabilities.  Thus,  it  appears  the  IIDA  Icing  and  Cloud  Tops 
components  may  be  giving  forecasters  an  enhanced  capability. 

Objective:  What  components  could  be  added  to  improve  algorithm  performance? 

AWC  area  forecasters  recommended  the  inclusion  of  vertical  motion  and  stability  fields  as 
possible  components  to  improve  IIDA  performance.  In  addition,  it  was  recommended  that 
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moisture  fields  from  the  Eta  and  Meso-Eta  models  be  considered,  since  forecasters  perceive 
these  fields  to  be  superior  to  the  Rapid  Update  Cycle  model,  60  km  version  (RUC)  moisture 
fields.  However,  it  should  be  noted  that  the  recent  implementation  of  the  Rapid  Update  Cycle 
model,  40  km  version  (RUC-2)  should  result  in  improved  moisture  fields  compared  to  those  with 
the  original  RUC. 

While  not  directly  related  to  performance,  AWC  area  forecasters  identified  the  need  for 
forecast  and  icing  intensity  components  in  order  to  make  IIDA  beneficial  for  operational  use. 

Objective:  Under  what  situations  does  the  IIDA  perform  well  or  not  perform  well? 

The  IIDA  performed  well  when  aircraft  icing  was  the  result  of  very  organized,  synoptic-scale 
weather  systems.  However,  many  other  sources  already  available  to  AWC  area  forecasters 
(e.g.,  National  Center  for  Atmospheric  Research,  Research  Applications  Program  (NCAR/RAP) 
algorithm)  also  perform  well  in  this  situation.  Thus,  the  IIDA  does  not  appear  to  be  an 
enhanced  capability  as  far  as  performance  is  concerned. 

On  some  instances,  there  was  the  identification  of  IIDA  performing  better  than  existing  sources 
when  low-level  icing  occurred.  However,  several  forecasters  noted  the  opposite  result, 
specifically  identifying  low-level  icing  situations  as  instances  of  when  IIDA  did  not  perform  well. 

It  was  noted  that  in  some  of  these  situations,  IIDA  identified  icing  when  the  cloud  tops  were 
1 ,000  to  2,000  feet  below  the  freezing  level.  In  addition  to  the  mixed  results  with  low-level  icing, 
an  instance  with  layered  clouds  was  identified  as  a  situation  where  IIDA  did  not  perform  well. 

Forecasters  noted  that  additional  testing  over  a  longer  period  of  time  and  during  the  most 
intense  part  of  the  winter  season  would  be  beneficial  in  further  identifying  strengths  and 
weaknesses  of  IIDA. 

Objective:  Subjectively  identify  how  the  IIDA  performance  compares  to  current  methods  used 
by  AWC  forecasters  for  identifying  the  presence  of  aircraft  icing  conditions. 

Overall,  AWC  area  forecasters  reported  that  IIDA  was  no  better  than  existing  sources  of 
available  icing  information.  It  was  noted  that  IIDA  was  similar  to  existing  products  and  did  not 
provide  additional  information.  Area  coverage  of  the  IIDA  was  similar  to  existing  products  with 
the  identification  of  large  areas  of  potential  aircraft  icing.  The  exception  to  this  was  the 
Supercooled  Large  Drop  (SLD)  Potential,  which  tended  to  identify  small  areas  of  potential  icing. 
Forecasters  perceived  the  SLD  Potential  as  performing  poorly,  however,  the  performance  may 
be  a  function  of  AIRMET  and  SIGMET  criteria,  which  focuses  upon  large  areas  of  potential 
icing  rather  than  specific  smaller  areas. 

The  performance  of  the  Icing  Potential  product,  measured  by  comparison  to  the  highest 
concentration  of  moderate  or  greater  PIREPs,  exhibited  positive  results  with  PIREPs 
corresponding  to  the  higher  values  of  the  product.  Existing  sources  of  icing  information  were 
not  compared  to  the  concentrations  of  moderate  or  greater  PIREPs,  thus  a  direct  comparison  of 
product  performance  is  not  available.  However,  based  upon  forecasters’  responses  throughout 
the  assessment  (e.g.,  see  sections  5.3.2,  5.3.4,  and  5.4.8),  it  appears  that  IIDA  is  as  good  as 
existing  sources  of  icing  information,  but  not  necessarily  better. 
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Objective:  Identify  AWC  data  inputs  and  platforms  so  that  further  development  can  be  tailored 
to  A  WC  operations. 

NCAR  personnel  accomplished  this  objective  during  the  implementation  of  IIDA  at  AWC. 
ACT-320  personnel  did  not  participate  in  this  activity,  thus  no  information  on  data  inputs  and 
platforms  is  included  in  this  report. 

6.2  OTHER  CONCLUSIONS. 

In  addition  to  the  assessment  objectives,  other  issues  identified  during  the  assessment  are 
discussed  in  the  following  sections. 

6.2.1  SLD  Performance. 

For  this  assessment,  the  SLD  Potential  product  and  associated  components  (i.e.,  SLD  Bases 
and  Tops)  did  not  appear  to  perform  well.  Product  rankings  placed  the  product  and 
components  low  in  regards  to  the  value  in  determining  Icing  AIRMETs  and  SIGMETs.  The 
performance  of  the  SLD  Potential  product  compared  to  the  highest  concentrations  of  moderate 
or  greater  PIREPs  indicated  that  the  product  did  not  identify  the  occurrence  of  aircraft  icing. 
Overall  utility  and  accuracy  ratings  were  negative.  However,  it  should  be  noted  that  in  the 
product  rankings  (see  section  5.2.1),  existing  sources  of  information  designed  to  identify  SLD 
related  icing  (i.e.,  the  Stovepipe  Algorithm  and  surface  observations  of  freezing  precipitation) 
received  low  rankings  in  regards  to  their  value  in  determining  AIRMETs  and  SIGMETs.  Thus, 
the  poor  overall  results  of  the  IIDA  SLD  Potential  product  and  components  may  be  indicative  of 
a  larger  issue  concerning  SLD  and  the  forecasting  of  hazardous  icing  situations  via  AIRMETs 
and  SIGMETs. 

The  assessment  results  are  not  sufficient  to  provide  reasons  why  the  SLD  performance  may 
have  been  poor.  Possible  reasons  include  that  the  product  and  components  are  inaccurate; 
there  was  a  lack  of  SLD  conditions  during  the  assessment  period;  or  adequate  training  and 
education  has  not  been  conducted  in  regards  to  potential  icing  threats  associated  with  SLD 
conditions.  Poor  SLD  performance  may  be  due  to  any  of  these  possible  reasons  or  additional 
factors  that  were  not  identified. 

In  regards  to  the  SLD  Potential  performance  as  measured  by  PIREPs  (see  section  5.2.2),  it  is 
possible  that  SLD  conditions  were  not  existing  in  the  atmosphere  during  the  assessment  phase. 
Using  moderate  or  greater  PIREPs  as  a  measure  of  performance  may  not  accurately  reflect  the 
performance  of  the  product  since  non-SLD  conditions  may  contribute  to  moderate  or  greater 
PIREPs.  In  addition,  the  “poor”  performance  of  the  SLD  Potential  in  identifying  only  small  areas 
of  icing  potential  (e.g.,  see  section  5.3.5)  may  actually  be  an  accurate  portrayal  of  SLD 
conditions  in  the  atmosphere. 

6.2.2  Bases  and  Tods. 

While  no  identifiable  trend  resulted  from  the  assessment  results,  there  appear  to  be  issues  with 
the  accuracy  of  Icing  and  Cloud  Bases  and  Tops.  Some  users  noted  that  tops  tended  to  be  too 
high  (by  4,000  to  6,000  feet  as  noted  by  one  forecaster).  Conversely,  other  users  noted  that 
the  Cloud  Tops  appeared  low  and  the  Icing  Bases  and  Tops  both  tended  to  have  a  low  bias. 
Accuracy  ratings  for  Icing  Bases  and  Tops  (described  in  section  5.4.4)  indicate  a  need  for 
improvement.  However,  Cloud  Bases  and  Tops  received  overall  positive  ratings  in  regards  to 
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accuracy,  especially  Cloud  Tops  (see  section  5.4.6).  Since  Icing  Tops  and  Cloud  Tops  both 
received  high  utility  ratings,  accuracy  improvements  may  provide  forecasters  an  effective 
source  of  icing  and  cloud  top  information. 

6.2.3  Icing  Type. 

The  Icing  Type  product  was  perceived  by  AWC  area  forecasters  as  performing  poorly.  At  least 
three  forecasters  noted  that  Icing  Type  was  not  applicable  or  useful.  One  forecaster 
commented  on  being  unaware  of  its  existence.  If  Icing  Type  continues  to  be  a  requirement  in 
AIRMETs  and  SIGMETs,  then  improvements  in  the  product  must  be  accomplished. 

6.2.4  Display  Issues. 

While  it  was  not  the  purpose  of  the  assessment  to  evaluate  display  concepts,  AWC  area 
forecasters  identified  several  display  issues.  These  issues  should  be  considered  if  the  IIDA  or 
other  similar  type  products  are  implemented  into  AWC  operations.  Issues  to  be  considered 
include: 

a.  The  blockiness  of  the  output  should  be  smoothed.  Note  that  AIRMETs  and  SIGMETs 
are  general  guidance  products,  thus  the  detail  provided  by  the  IIDA  may  not  be  required  for 
AIRMET  and  SIGMET  generation.  However,  if  finer  detailed  products  are  needed,  then 
smoothing  of  the  output  may  not  be  desirable. 

b.  Only  Icing  Potential  output  above  a  certain  threshold  should  be  displayed.  The  display 
used  in  the  IIDA  Assessment  displayed  all  output  in  the  range  of  0-100.  Thresholding  could 
reduce  the  apparent  overforecasting  noted  by  forecasters,  especially  since  the  greatest 
concentration  of  moderate  or  greater  PIREPs  tended  to  occur  with  Icing  Potential  values  of 
greater  than  60. 

c.  Forecasters  favored  integration  and  additional  integration  should  be  considered.  For 
example,  it  was  noted  that  a  symbol  could  be  placed  on  the  potential  products  identifying  bases 
and  tops. 

d.  Removing  the  display  of  icing  PIREPs  due  to  convective  activity,  since  Icing  AIRMETs 
and  SIGMETs  are  not  required  for  icing  resulting  from  convection. 

6.2.5  Reliability. 

Many  comments  were  made  during  the  assessment  concerning  the  reliability  and  availability  of 
IIDA  products  and  components.  While  round-the-clock  availability  of  IIDA  should  not  have  been 
expected  (due  to  the  fact  that  National  Center  for  Atmospheric  Research  (NCAR)  is  not  an 
operational  facility  with  a  24-hour  per  day  support  staff),  the  lack  of  availability  potentially  biased 
several  forecasters  against  IIDA.  Consideration  should  be  given  in  future  assessments  and 
evaluations  to  having  algorithms  and  products  running  at  AWC  where  availability  can  be 
ensured. 
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6.2.6  AWC  Operations. 


The  results  of  the  IIDA  Assessment  identified  several  issues  concerning  AWC  area  forecaster 
operations.  These  issues  should  be  considered  for  future  assessments  of  any  products,  not 
just  icing-related  ones. 

a.  AWC  products  are  heavily  influenced  by  observations.  As  seen  in  the  IIDA  Assessment 
results,  the  products  that  have  the  most  value  in  determining  AIRMETs  and  SIGMETs  are 
observations.  Derived  products  become  more  valuable  only  when  observations  are  not 
available. 

b.  AWC  is  a  relatively  data  rich  forecast  environment  with  support  staff  developing 
products  of  their  own.  Forecasters  have  a  multitude  of  products  to  choose  from  in  order  to 
assist  in  the  formation  of  aviation  weather  products.  Due  to  the  amount  of  existing  products, 
new  products  introduced  for  assessment  or  evaluation  may  receive  lower  ratings  than  if  the 
product  was  provided  for  other  operational  environments  (e.g.,  Center  Weather  Service  Units 
(CWSU)  and  airline  meteorology  departments). 

c.  AWC  area  forecasters  have  significant  experience  in  the  development  of  aviation 
weather  hazard  products.  Consequently,  forecasters  are  reluctant  to  deviate  from  established 
procedures  and  products,  which  have  demonstrated  performance  levels  resulting  in  forecaster 
confidence.  New  products  introduced  to  the  forecaster  environment  must  have  some 
demonstrable  means  of  performance  in  order  for  forecasters  to  be  willing  to  review  and/or  use 
the  products. 

7.  RECOMMENDATIONS. 

The  purpose  of  the  Integrated  Icing  Diagnostic  Algorithm  (IIDA)  Assessment  was  to  evaluate 
the  utility  of  the  IIDA  products  and  components  and  to  provide  feedback  to  the  National  Center 
for  Atmospheric  Research/Research  Applications  Program  (NCAR/RAP)  for  further 
development.  In  light  of  this  purpose,  the  following  recommendations  are  made: 

a.  Future  emphasis  should  be  on  the  development  of  a  forecast  capability  that  also 
incorporates  icing  intensity. 

b.  Consideration  should  be  given  to  incorporating  vertical  velocity  and  stability  fields  into 
IIDA. 

c.  Consideration  should  be  given  to  producing  output  using  the  Eta  or  Meso-Eta  models 
due  to  the  perceived  superiority  of  their  moisture  fields.  However,  Rapid  Update  Cycle  model, 
40  kilometer  (km)  version  (RUC-2)  fields  may  have  alleviated  the  potential  shortcomings  in  the 
original  Rapid  Update  Cycle  model  60  km  version  (RUC). 

d.  NCAR/RAP  needs  to  attempt  to  identify  the  reasons  for  the  poor  performance  of 
Supercooled  Large  Drop  (SLD)  products  and  components.  At  a  minimum,  research  results 
concerning  SLD-related  icing  and  meteorological  parameters  (such  as  freezing  precipitation) 
need  to  be  provided  to  operational  forecasters  through  training  and  education  opportunities. 
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e.  Consideration  needs  to  be  given  to  conducting  future  assessments  and  evaluations  at 
locations  other  than  Aviation  Weather  Center  (AWC),  such  as  Center  Weather  Service  Units 
(CWSUs)  and  commercial  airlines. 

f.  All  assessment,  demonstration,  and  evaluation  activities  must  be  accompanied  by 
adequate  training.  Failure  to  provide  adequate  training  can  potentially  influence  results. 

8.  ACRONYMS. 


ACT-320 

AIRMET 

AWC 

CAPPI 

CWSU 

EPV 

FAA 

GOES 

IIDA 

km 

mb 

N-AWIPS 

NCAR 

NCEP 

NEXRAD 

NIDS 

NNICE 

NWS 

PC 

PIREP 

RAOB 

RAP 

RH 

RUC 

RUC-2 

SIGMET 

SLD 

T 

2- D 

3- D 


FAA  Technical  Center  Weather  Branch 

Airmens  Meteorological  Statement 

Aviation  Weather  Center 

Constant  Altitude  Plan  Position  Indicator 

Center  Weather  Service  Unit 

Equivalent  Potential  Vorticity 

Federal  Aviation  Administration 

Geostationary  Orbiting  Environmental  Satellite 

Integrated  Icing  Diagnostic  Algorithm 

kilometer 

millibars 

Advanced  Weather  Interactive  Processing  System  for  National  Centers 

National  Center  for  Atmospheric  Research 

National  Centers  for  Environmental  Prediction 

Next  Generation  Weather  Radar 

NEXRAD  Information  Dissemination  Service 

Neural  Net  Icing  Product 

National  Weather  Service 

Personal  Computer 

Pilot  Report 

Rawinsonde  Observation 
Research  Applications  Program 
Relative  Humidity 

Rapid  Update  Cycle  model,  60  km  version 

Rapid  Update  Cycle  model,  40  km  version 

Significant  Meteorological  Statement 

Supercooled  Large  Drop 

temperature 

two-dimensional 

three-dimensional 
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APPENDIX  A 


DAILY  QUESTIONNAIRE 


APPENDIX  A 
DAILY  QUESTIONNAIRE 


*  UDA  Daily  Forcaster  Log 


1 .  What  was  the  predominant  meteorological  cause  of  icing  in  your  area  of  responsibility?  _ _ 

C  Synoptic  system  C  Terrain  induced  C  Convection  C  Other. 

2.  The  IIDA  Icing  Potential  presents  a  range  from  0*100.  Select  the  Icing  Potential  range  where  the  greatest  concentration  of 

Moderate  or  greater  icing  pireps  occurred:  _ _ _ 

r  No  Pireps  available  C  IIDA  did  not  identify  C  0-20  C  20  -  40  C  40  *  60  C  60  -  80  C  80  - 1 00 

3  The  SLD  Potential  presents  a  range  from  0-100.  Select  the  SLD  Potential  range  where  the  greatest  concentration  of  Moderate  or 
greater  icing  pireps  occurred:  _ _ _ - 


C  No  Pireps  available  C  IIDA  did  not  identify  C  0-20 


r  20-40  r  40-60  r  60-80  r  80-100 


What  value  did  products  provide  in  determining  your  Icing  Airmet(s)/$igmet(s).  Rate  the  following  products  by  selecting  the  appropriate 
rating  according  to  your  use  today  in  determining  your  I cing  Airmet(s)/Sigmetfs). 


1.  1IDA  Icing  Potential 

r 

r 

r 

r 

r 

2.  IIDA  SLD  Potential 

r 

r 

r 

r 

r 

3.  IIDA  Icing  Type 

r 

r 

r 

r 

r 

A.  IIDA  Icing  bases  and  tops 

r 

r 

r 

r 

r 

5.  IIDA  SLD  bases  and  tops 

r 

r 

r 

r 

r 

6.  IIDA  Cloud  bases  and  tops 

r 

r 

r 

r 

r 

7.  Graphical  Pireps 

r 

r 

r 

r 

r 

8.  Text  Pireps 

r 

r 

r 

r 

r 

9.  Existing 

r 

r 

r 

r 

r 

10.  Freezing  precip  observations 

r 

r 

r 

r 

r 

11.  Raob  soundings 

r 

r 

r 

r 

r 

A-2 


IHBEI 


What  value  did  products  provide  in  determining your  Icing  Airmet($)/Sigmet(s).  Rate  the  following  products  by  selecting  the  appropriate 
rating  according  to  your  use  today  in  determining  your  Icing  Armetfs)/Sigmet(s), 


1 2.  Satellite 

r 

r 

r 

r 

r 

13.  RADAR  reflectivity 

r 

r 

r 

r 

r 

14.  CAP! 

r 

r 

r 

r 

r 

1 5.  Model  soundings  of  T  and  RH 

r 

r 

r 

r 

r 

16.  Model  soundings  of  EPV 

r 

r 

r 

r 

r 

17.  Neural  Net  Icing  (NNice) 

r 

r 

r 

r 

r 

18.  NCAR/RAP  Icing 

r 

r 

r 

r 

r 

19.  Stovepipe  Algorithm 

r 

r 

r 

r 

r 

20.  Other:  |  " 

r 

r 

r 

r 

r 

21.  Other:  | 

r 

r 

r 

r 

r 

22.  Other  [~  “ 

r 

r 

r 

r 

r 

Clear  Product  #20  Clear  Product  #21  Clear  Product  #22 


23.  Did  the  1 1  DA  add  anything  to  your  knowledge  of  today’s  icing  situation  that  could  not  be  readily  perceived  from  products  already 
available:  r  Yes  C  No 


Please  comment 
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END-OF-ASSESSMENT  QUESTIONNAIRE 


1.  Ichg  Potential 


a.  Utity 

r 

r 

r 

r 

r 

r 

b.  Refiabity 

r 

r 

r 

r 

r 

r 

c.  Accuracy 

r 

r 

r 

r 

r 

r 

2  SLD  Potential 

a.  Utity 

r 

r 

r 

r 

r 

r 

h.  Re&ab&y 

r 

r 

r 

r 

r 

r 

tx  Accuracy 

& 

r 

r 

r 

r 

r 

3.  Icing  Type 

a.  Utity 

r 

r 

r 

r 

r 

r 

b.  Reliability 

r 

r 

r 

r 

r 

r 

c.  Accuracy 

r 

r 

r 

r 

r 

r 

4.  List  suggestions  lor  improving  any  of  the  above  products.  Please  be  sue  to  include  the  product  name  in  the  suggestions). 


5.  List  other  products  that  you  think  should  be  added  to  the  (IDA, 


Next 


Rating  Scale 
Definitions 
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6.  Cloud  Base  Height 


a  Utility 

r 

r 

r 

r 

b.  Refiabity 

r 

r 

r 

r 

a  Accuracy 

r 

r 

r 

r 

7.  Qoud  Top  Height 

a  Utility 

r 

r 

r 

r 

b-  RefiabKy 

r 

r 

r 

r 

a  Accuracy 

r 

r 

r 

r 

8.  Icong  Bases 

a.  UtiBty 

r 

r 

r 

r 

b.  Reliabilit y 

r 

r 

r 

r 

c.  Accuracy 

r 

r 

r 

r 

9.  Icing  Tops 

a  Utility 

r 

r 

r 

r 

b.  Reiabity 

r 

r 

r 

r 

c.  Accuracy 

r 

r 

r 

r 

10.  SLD  Bases 

a  Ut#ty 

r 

r 

r 

r 

b.  Reiabity 

r 

r 

r 

r 

c.  Accuracy 

r 

r 

r 

r 

Next 

Rating  Scale 

Definitions 
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IT.  SLD  Tops 

a  ufifty  r  r  r  r  r 

b.  Refiabftfty  C  C  C  C  C 

c.  Accuracy  C  C*  &  C  C* 

12.  Ust  suggestions  for  improving  ary  erf  the  Please  be  sure  to  include  the  compcr>ent  rrame  in  the  suggestions), 

I  3 

zl 


13.  Does  the  integrated  product  provide  any  value  beyond  having  the  individual  components  by  themselves?  C*  Yes  C  No 
Please  explain.  Provide  examples  if  necessaiy. 


uc 
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APPENDIX  C 

FORECASTER  COMMENTS  FROM  THE  DAILY  QUESTIONNAIRE 


Question  -  Did  the  IIDA  add  anything  to  your  knowledge  of  today’s  icing  situation  that 
could  not  be  readily  percieved  from  products  already  available? 

iida  did  not  show  any  clouds  ovr  sern  tx  where  tops  were  low  (around  5,000  ft),  it  could  have 
been  useful  in  helping  to  determine  the  cld  tops  there. 

iida  did  pretty  good  job  showing  areas  of  icing,  but  all  other  diagnostics  and  algorithms  did  also. 

neb/ks/wm  mo  aea  of  icg  had  tops  amd  10,000.  Hier  toped  elds  fooled  iida  into  putting  too  high 
tops  in  this  area.  Yet  low  topped  icg  080  sm  Iwr  mi  was  correctly  shown  despite  thick  ci.  iida 
showed  low  topped  icg  area  over  sm  Iwr  mi  better  than  rap/nnice.  tops  were  good  in  this  area 

low  elds  based  at  035  tops  060  from  ar  thru  tn/ky.  iida  correctly  showed  a  narrow  column 
below  060  of  low  to  middle  potential  icing-  that  was  good,  but  cloud  top  algorithm  was  wacky, 
showing  cld  tops  to  140 

ncar/rap,  nnice  both  showed  very  large  area  of  icg  ovr  cntrl  plains/mid  ms  vly  where 
convectively  produced  clouds  were  increasing...virtually  no  icing  in  that  area  however. 

icing  and  cloud  tops  in  the  ky/tn  area  were  120-130.  iida  once  again  showed  too  high  of  tops 
(160)  . 

iida  basicly  showed  tha  same  area  II  aready  had  sketched  out. 

iida  generally  showed  what  1  aready  had  sketched  out  for  my  icing  areas,  iida  overforecasted 
icg  ovr  AZ  where  there  was  no  elds. 

overforecast  over  UT/CO  /MT/WY  basicly  showing  icing  in  cirrus  clouds. 

IIDA  covered  just  about  the  same  area  1  analyzed  from  other  sources  in  making  my  icing 
airmets.  1  got  the  same  info  from  other  sources.  This  time  it  did  not  have  large  overforecasted 
areas  where  there  were  nlittle  or  no  clouds. 

little  value  was  added  and  most  of  the  reports  were  not  in  the  higher  potential  areas. 

large  area  few  reports,  clouds  over  area  but  tat  will  not  help  define  area  verry  well. 

fair  location  for  general  icing  but  sld  not  good. 

all  pireps  of  icg  Igt  to  mod  or  gtr  were  of  rime  icg  today. 

way  to  low  on  bases  and  way  to  far  south. 

No  icing  threat  anticipated  other  than  convection.  Did  mention  isolated  light  to  moderate 

ND?MN  based  on  satellite,  neural  network  and  IIDA 

satellite  imagery,  Pireps  and  Neural  Network  aided  in  identifeation  of  threat  areas.  IIDA  also 
supported  these  areas  but  added  nothing  that  could  not  have  been  determined  from  the  above 
sources. 

IIDA  broken.  It  has  not  updated  since  2PM  Saturday  March  28 

IIDA  highlighted  similar  area  as  Neural  Net  and  NCAR/RAP 

Neural  Network,  satellite  imagery  and  PIREPS  identified  icing  area 

Due  to  N-AWIPS  problem,  IIDA  did  not  update  after  03Z 

All  icing  appeared  to  be  related  to  convection.  Icing  Airmet  said 11  No  sgfnt  ice  exp  outside 
cnvtv  actvty.” 

Ice  was  associated  with  a  large  scale,  slow  moving  low.  There  was  a  high  degree  of 
agreement  among  the  various  diagnostic  tools  and  among  various  forecast  tools. 

Fairly  good  agreement  among  diagnostics  and  forecasts. 

NCAR/RAP  was  out  to  lunch  and  IIDA  helped  to  identify  that. 

Was  similar  to  NNICE  and  helped  in  decision  making  since  NCAR/RAP  and  NNICE  differed  in 
coverage  in  a  particular  area. 
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Very  Isolated  ice  reports  today  mainly  in  cirrus  which  is  not  forecast  well  by  any  model. 

Received  only  2  reports  which  lead  me  to  not  issue  any  airmets  for  ice.  Any  other  ice  expected 
to  be  convectively  related. 

Several  light  ice  reportswith  one  or  two  isolated  moderate  reports  in  comma  head/baroclinic 
zone  from  Nebraska  to  Wisconsin  that  occurred  in  area  where  IIDA  indicated  no  icing.  Little  or 
no  precip  occuring  at  surface. 

Most  all  reports  associated  with  convection  today  and  very  little/no  icing  where  IIDA  showed 
large  values 

Typical  synoptic  type  system.  IIDA  underdid  tops  of  icing  in  Illinois  as  call  from  ZKC  indicated 
all  aircraft  reporting  icing  between  120  and  FL205  between  DEC  and  VLA  and  tops  of  icing  on 
IIDA  only  140. 

Did  a  good  job  on  depicting  low  level  icing  situation  over  Nebraska  and  Iowa.  Only  product  1 
saw  which  depicted  it  However  overdid  icing  in  other  areas. 

No  iceing  reports,  although  soundings  and  ncar/rap  forecast  iceing. 

other  products  showed  ice  info  just  as  wello. 

iida  product  helped  to  confirm  locations  of  ice  and  tops  and  baseoo 

Other  products  such  as  NCAR  RAP,  satellite  data  and  NAWIPS  plots  of  icing  reports  were 
sufficient  to  issue  a  forecast  of  icing  this  morning. 

Products  such  as  NCAR/RAP  icing,  satellite  and  plots  of  icing  pireps  overlayed  on  satellite 
image  were  of  greatest  value. 

No  added  value  over  small  area  of  interest  in  New  England. 

display  a  little  too  noisy,  values  below  40%  should  be  eliminated 

Portions  of  the  iida  graphics  had  missing  data. 

Missed  on  two  sev  and  one  mod  icg  report  at  FL160-FL220 
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FORECASTER  INTERVIEW  RESPONSES 


The  following  responses  are  paraphrased  based  upon  AWC  area  forecaster  responses  to  the 
structured  interview  questions. 


QUESTION  1  -  Did  you  use  the  IIDA  for  producing  Icing  AIRMETs/SIGMETs?  Please 
explain  why  or  why  nok  _ 

Did  not  add  much.  Did  not  really  use  for  SIGMET  generation.  Already  have  enough 
information.  I  am  used  to  my  old  ways  -what  I’ve  used  for  the  past  22  years. 


Did  not  use  it  at  all.  Became  familiar  with  it  a  few  days  ago. 


Looked  at  it  twice.  Use  Pilot  reports.  Surface  based  information  gives  a  better  idea  of  what  is 
going  on.  I  wonder  about  the  accuracy  of  the  product  in  the  West.  When  you  have  moisture 
and  movement  there’s  icing. 


Not  on  shift  to  use  it.  Looked  at  it  but  it  was  down  since  Sunday.  Still,  would  probaby  not  have 
used  it  for  icing  AIRMETs. 


Used  very  little  since  I  am  a  new  FA  and  getting  familiar  with  other  systems.  Looked  at  it  for  a 
couple  of  weeks.  Sometimes  would  look  at  it  before  issuance  of  an  AIRMET  to  verify  what  was 
issued.  Availability  is  a  problem. 


One  tool  looked  at  for  diagnostics.  Already  have  an  idea  where  icing  is  and  where  it  is  going.  I 
compare  to  what  is  really  happening.  If  it  looks  good  I  would  issue  an  AIRMET  for  that  area. 

It  confirmed  icing  at  a  particular  moment. 


Used  IIDA  for  AIRMETS,  not  for  SIGMETS.  However,  used  it  extensively  when  it  was  available 
on  the  web  site. 


Used  IIDA  around  2/3  of  the  time.  Did  not  use  it  for  SIGMET  generation.  Compared  to  other 
products,  it  captured  general  icing  well. 


Used  IIDA  as  part  of  the  overall  icing  process.  It  was  an  added  component  but  since  it  was 
new  and  experimental,  I  did  what  I  usually  did.  Found  it  most  useful  during  the  midnight  shift 
when  there  were  no  pilot  reports.  It  did  give  an  indication  of  where  icing  was  starting. 


I  look  at  it  mainly  for  Icing  AIRMETs.  It  is  not  the  sole  source  of  information,  but  one  source. 
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QUESTION  2  ■  How  did  IIDA  information  compare  to  other  icing  information  sources? 

Looks  at  RAOBs  first,  then  heights  and  types  from  the  ice  map  (at  Oz  and  12z).  Compare  to 
neural  net  for  percentage  of  icing  in  the  layers.  Use  diagnostic  products  to  get  the  basics. 
Need  to  take  convective  information  out  for  a  true  report.  Also  look  at  CWSU  reports  -  icing  up 
to  30,000  ft.  Sensors  on  aircraft  are  poor  and  sometimes  not  a  viable  measure.  Other 
products  used  include:  Vertical  motion  fields  -  vertical  profile  potential  of  vertical  motion  on  the 
NGM.  Neural  Net  is  the  preferred  icing  product.  Stovepipe  is  similar  to  the  IIDA  -  and  was 

used  without  much  success.  Stovepipe  is  rarely  used. _ 

Seems  similar  to  the  Stovepipe  product.  It  looked  better  than  Stovepipe  but  I  don’t  use 
Stovepipe.  I  need  a  forecast  -this  is  only  a  detection  product.  Mostly  use  PIREPS,  the  RAP 
product  or  Neural  Net.  Disappointed  with  RUC  -  it  doesn’t  go  out  far  enough.  Prefer  NGM 

and  Meso  Eta  models. _ _ 

Did  not  compare  -  seemed  useless  -  no  forecast  in  it.  I  like  RUC  II  and  the  Meso  Eta  model 
over  4  panels.  There’s  too  much  information  on  the  RUC  II  model.  Looks  at  50%  and  higher 
levels  on  RUC  II  (blue  areas)  Then  all  bases,  then  bases  and  tops.  Works  well  for  all  sections 
of  the  country.  For  SIGMETS,  mainly  use  Pireps,  surface  reports,  visibility  and  intensity. 

No  better  than  NCAR/RAP  (a  little  better). _ 

IIDA  missed  icing  above  16,000  ft.  due  to  the  mountains.  Use  the  Meso  ETA  model  where 
90%  relative  humidity  and  temperatures  were  bordering  at  the  higher  edge  of  the  icing 

spectrum.  Also  use  vertical  motion.  _ 

Good  diagnostic  tool  like  Neural  Net  -  most  icing  appears  where  it’s  red.  Icing  bases  and  tops 
are  not  useful.  Uses  the  Meso  Eta  4  times  a  day.  However,  questions  the  viability  of  the 
RUC  2.  Finds  that  the  PROF.  ICG  product  which  includes  temperature,  RH,  and  forecast 
soundings  (EPV)  is  good. _ 

Looked  a  lot  like  several  broadbrush  products,  i.e.:  RH  and  thickness  and  the  NCAR/RAP 

product  seemed  to  have  similar  output.  _ 

Compared  IIDA  with  ETA,  Meso  ETA,  Neural  Net,  and  RUC.  General  icing  seemed  OK  - 
bases  and  tops  were  good  and  pretty  accurate.  Other  products  showed  much  the  same. 
Colors  were  hard  to  interpret  and  needed  to  be  smoothed  out.  The  number  of  colors  was 
adequate  but  contouring  also  needs  to  be  smoothed.  It  did  well  with  layered  clouds. 

Coverage  was  good,  but  sometimes  overdone. _ 

Liked  the  overlaid  PIREPs  but  found  them  difficult  to  read.  Didn’t  compare  to  other  sources. 
Found  it  simpler  to  use  than  profiler  information.  Found  it  somewhat  reliable  and  began  to 

develop  trust  in  it.  May  use  IIDA  output  for  AIRMETs  but  probably  not  for  SIGMETs. _ 

The  display  is  too  busy.  The  colors  are  overwhelming.  The  thresholds  need  to  be  established. 
Its  okay  if  PIREPs  fall  outside  of  an  area.  Colors  need  to  be  changed  to  pick  out  the  PIREPSs. 
Probably  the  last  source  used  for  issuing  AIRMETs.  Prefers  other  sources  and  the  ability  to 
window  four  displays  at  once. _ 
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Icing  intensity  may  be  most  useful.  Icing  type  could  be  useful. _ 

Wasn’t  worth  the  time  -  didn’t  look  at  it.  Terrain  is  most  important. _ 

SLD  is  the  worst  component.  SLD  does  not  get  to  what  you  want.  Freezing  rain  is  not  as 

important.  Use  for  product  currently:  1 )  define  process,  and  2)  run  and  use  the  process. _ 

Clouds  tops  and  bases,  and  icing  layers  (potential). 

Did  not  use  SLD,  especially  for  the  Western  region. _ 

Icing  potential  was  the  only  component  really  used.  Icing  bases  and  tops  were  not  used  much. 
Was  not  comfortable  using  the  SLD  component.  Would  probably  use  and  look  at  the  IIDA  more 

throughout  the  winter.  Does  not  know  if  "Icing  Type"  is  needed  or  not. _ 

Icing  tops  were  less  useful  than  expected.  IIDA  reported  very  high  tops,  i.e.  cloud  tops  at 
12,000  ft.  were  reported  at  16,000  ft.  Although  cloud  heights  were  well  established,  IIDA  would 

typically  report  them  at  4-6  thousand  ft.  too  high.  Cloud  bases,  however,  were  good. _ 

Looked  at  icing  bases  and  tops  more  than  clouds.  Icing  bases  and  tops  did  a  good  job.  SLD 
bases  and  tops  were  not  useful.  Icing  probability  was  used  the  most  and  found  to  be  most 

useful.  Did  not  use  icing  type  and  didn’t  know  it  was  available. _ _ 

Icing  potential  and  icing  layers  were  most  useful.  Sometimes  bases  and  tops  information  was 
useful.  Icing  type  had  some  merit. 

Components  not  needed  include:  Clouds  information;  SLD  (which  gives  some  good 
information  but  to  a  limited  extent)  SLD  information,  however,  is  probably  underutilized. 

There’s  not  enough  time  to  include  this  information  in  an  AIRMET. _ 

Mostly  used  Icing  potential.  Received  good  ideas  on  tops  from  PIREPs  and  got  a  handle  on 
where  icing  would  be.  Freezing  levels  are  obtained  from  surface  charts  and  satellite  imagary. 
Likes  to  see  a  general  pattern  of  icing  potential  and  then  uses  models  to  see  how  this  will 
change.  _ 


QUESTION  4  -  Describe  typical  situations,  if  any,  where  the  algorithm  and/or  its 
components  performed  well. _ 

Yes,  there  were  a  couple  of  instances  where  the  product  performed  well  -  appeared  to  match 

some  of  the  other  output. _ 

No  comment. _ 

For  other  FA  positions  it  possibly  could  be  useful. _ 

Icing  potential  performs  better.  Clouds  information  is  the  foundation  in  combination  with 

observation  and  and  satellite  data. _ 

Liked  the  graphical  display. _ 

IIDA  is  as  good  as  CAPI  if  icing  levels  are  in  the  moderate  to  high  range.  Worked  best  with 
deep  area  moisture  around  a  low  pressure  area  (like  other  models).  Would  identify  a  well 

organized  system. _ 

"Told  me  what  I  already  knew".  Icing  potential  could  show  how  icing  is  trending,  however,  this 
information  is  redundant.  It  would  be  helpful  if  it  were  compared  to  the  NCAR/RAP  product 

to  see  where  ice  is  detected  and  then  forecast  it  out. _ 

Can’t  recall  when  it  out-  performed  any  other  information  source.  Probably  as  good  as 
anything. _ 

In  large  scale  synoptic  situations  that  were  well  organized. _ 

Well  defined  synoptic  situation.  Covers  everything  -  almost  too  much.  Doesn’t  add  anything 
to  what  is  already  available.  However,  the  product  does  provide  the  initial  pattern. _ 
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QUESTION  5  -  Describe  typical  situations,  if  any,  where  the  algorithm  and/or  its 
components  performed  poorly.  _ 

Icing  type  and  SLD  don’t  seem  to  correlate. _ _ 

No  comment. _ 

In  the  West  there  were  no  SLD  areas.  Does  not  meet  AIRMET  criteria. _ 

SLD  performed  poorly.  Not  a  good  conceptual  model.  All  significant  icing  develops  in  range  of 
SLDs  -  limited  to  processes  -  other  processes  not  emphasized  enough.  Need  to  look  at  whole 
process  of  how  icing  forms.  Better  algorithms  emphasize  what  we  know  and  what  the 
atmosphere  is  saying  .  When  you  relate  the  algorithm  to  the  PIREPs,  IIDA  misses  the  mark. 
The  foundation  though  is  necessary.  We  need  to  develop  a  better  algorithm  on  top. 

Different  output  would  be  preferable.  Trying  to  show  SLD  and  icing  potential  obscures  the 
PIREPs.  Icing  type  is  totally  unnecessary  and  can  be  eliminated.  4  panel  plot  is  not  the  best 

display.  Colors  are  OK  however.  Avoid  long  down  times. _ 

Moderate  reports  missed  big  when  compared  to  PIREPs.  The  display  was  too  noisy.  High 
icing  potential  (red)  was  indicated  at  certain  altitudes,  but  there  was  no  verification.  Product 
was  “  overdoing"  detection.  In  Montana  and  Idaho,  there  was  a  problem  at  low  altitude  -  the 
product  was  underforecasting.  IIDA  is  very  time  consuming  to  use.  Reliability  is  a  problem. 
Colors  were  not  significant  except  for  red.  Everything  else  is  inconsequential  (on  icing 
potential).  Did  not  always  work  well  especially  when  there  was  lower  icing  under  10,000  ft.  - 
over  10,000  feet  it  did  a  good  job  like  the  models.  ETA,  NGM,  and  Meso  ETA  worked  best  for 
low  level  icing.  Tops  information  is  inadequate.  There  are  better  things  to  look  at.  However, 

when  looked  at,  there  was  not  much  icing  around.  Very  skeptical  of  the  SLD  component. _ 

Already  describes  the  NCAR/RAP  and  RH  +  temp  output.  Needs  a  differentiation  of  severity. 
The  category  and  severity  of  icing  needs  to  be  known.  IIDA  did  not  add  anything.  It  paints  large 
areas  that  were  already  identified.  Not  filtering  down  enough  -  areas  are  too  large,  i.e.,  high 
icing  potential  covered  a  very  big  area.  However,  it  missed  the  freezing  level  on  low  topped 
cloud  decks,  sometimes  by  1000  feet  -  this  is  critical.  The  top  of  the  cloud  deck  is  the  most 
likely  location  for  icing.  The  SLD  component  cuts  the  area  down  a  great  deal,  but  doesn’t  relate 

to  anything.  Need  to  test  IIDA  in  the  winter. _ 

Once  IIDA  totally  missed  significant  icing.  The  area  was  blatantly  apparent  -  a  synoptic  stable 

system  in  the  Pacific  North  West.  There  was  lots  of  ice,  but  the  algorithm  missed  it. _ 

Sometimes  there  were  software  glitches  and  sometimes  the  product  did  not  update. _ 

Sometimes  icing  PIREPs  fell  outside  of  a  detected  area  -  even  when  the  coverage  was 
extensive.  There  was  some  doubt  in  these  situations.  Its  nice  to  see  where  PIREPs  are  falling 
in  relation  to  everything  else. _ 
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QUESTION  6  -  Miscellaneous  Responses _ 

No  Comment. _ 

Needs  forecast  up  to  18  hours  for  outlook. _ 

No  comment. _ 

Threshold  would  be  good,  l.e.:  icing  below  50,000  ft  for  icing  potential.  Calibration  numbers 
are  questionable.  Concentrate  on  intensity  and  then  lead  the  way  in  how  icing  is  forecast. 
Would  like  a  layer  forecast  for  thicker  layers.  Getting  through  different  levels  is  time 
consuming.  Adding  vertical  motion  would  be  helpful.  Added  forecast  potential  would 
compliment  what  is  already  used.  The  forecast  does  not  need  to  go  beyond  12  hours.  Could 

be  used  for  SIGMETs. _ 

Wants  to  keep  all  of  the  IIDA  components  so  that  everyone  can  choose  individually.  Everyone 
has  different  needs.  Enhancements:  Cut  off  temperature  at  -25c  (which  is  the  AWC  cut-  off). 
Could  be  cause  of  discrepancies.  Also  clouds  looked  lower  than  they  actually  were.  Bases 
were  OK  for  freezing  level,  but  there  were  discrepancies  with  tops.  A  forecast  is  needed  since 
90%  of  time  you  need  to  be  looking  out.  It  would  be  good  if  it  could  forecast  out  to  12  hours. 
The  Neural  Net  product  forecasts  out  as  far  as  the  models.  If  a  forecast  is  reliable,  it  will  be 
compared  to  the  Neural  Net  and  the  best  one  wins.  However,  lateness  of  season  for  this 

product  provided  an  unfair  test. _ 

A  forecast  would  be  an  improvement,  but  by  definition  you  can’t  get  a  forecast  with 
observations,  satellite  and  radar.  The  following  are  not  needed:  Icing  type  -  No  better  than 
what  we  do.,  SLD  -  not  much  use,  however  there  may  be  some  marginal  use  in  colder 
weather.  A  better  way  is  needed  of  showing  multiple  layers  of  cloud  tops.  Perhaps  looping  the 
clouds  to  see  multiple  layers  would  give  unique  information.  Could  also  use  a  better  display  of 

clouds. _ 

May  use  IIDA  if  its  here  and  improvements  are  made.  Could  be  a  useful  diagnostic  tool  to 
use  with  other  weather  sources.  SLD  "goodness"  is  questionable  (like  the  Stovepipe  product.) 
Integration  of  the  products  on  IIDA  may  have  saved  some  time  (questionable)  but  in  a  situation 
when  there  is  no  time  to  look  at  all  models  and  soundings,  it  wouldn’t  be  used.  Could  be 

useful  for  comparisons. _ 

IIDA  Would  be  a  worthwhile  tool  overall,  added  to  everything  else. _ 

Time  savings?  Yes  as  far  as  the  pattern  it  shows.  Reliability  was  not  good.  Sometimes  the 
product  was  down  and  you  never  knew  when  it  would  be  available. 
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PRODUCT  IMPROVEMENTS  IDENTIFIED  FROM  THE 
END-OF-ASSESSMENT  QUESTIONNAIRE 


Question  -  List  suggestions  for  improving  any  of  the  products  (Icing  Potential,  SLD 
Potential,  Icing  Type).  Please  be  sure  to  include  the  product  name  in  the  suggestion(s). 

icg  potential  needs  to  be  related  to  intensity,  since  we  only  issue  airmets  for  icing  that  is 
expected  to  be  at  least  occasionally  moderate  or  greater,  the  iida’s  depiction  of  a  near  certainty 

of  what  will  surely  be  light  icing  has  little  utility. _ 

Vertical  motion  fields  need  to  be  included  such  as  72v.  I  have  found  when  mod  to  stg 
subsidence  or  mod  to  stg  upward  motions  are  occurring  then  the  amount  of  icing  is  lessened. 

The  map  value  of  +  or  -  3  seem  to  be  the  outside  limits  many  reports. _ 

Try  to  make  IIDA  more  reliable.  There  were  large  blocks  of  time  over  the  weekends  when  IIDA 
was  unavailable.  Try  to  incorporate  satellite  imagery  within  the  IIDA  algorithm.  There  were  time 

when  IIDA  depicted  icing  where  there  were  no  clouds. _ 

there  were  too  many  times  when  all  of  the  data  was  not  avialable. _ 

Did  not  see  much  difference  between  this  and  other  products.  Also  it  seemed  to  have  a  hard 
time  in  low  cloud  top  situations  handling  mid  level  clouds,  i  think  this  is  result  of  the  RUC  model 

which  has  RH  problems. _ 

We  need  forecasting,  not  hindcasting  aids. _ 

General  icing  and  SLD  categories  do  not  “hit"  the  target  which  is  icing  intensity.  General  icing 
was  too  much  of  a  temp/RH  cloud  diagnostic.  SLD  does  not  give  clues  to  higher  intensity  icing 

so  has  no  real  skill.  Icing  type  also  has  no  skill. _ 

icg  potential  product  was  pretty  good  in  locating  general  icg  areas  but  overforecasted  areas  at 
times,  smoothing  contours  would  help  in  analyzing  product,  sld  potential  product  has  little 

utility,  saw  no  pireps  to  indct  sld  icg. _ 

Perhaps  some  smoothing  technique  could  be  used  to  eliminate  the  extreme  blotchiness  of  the 

icing  potential. _ 

More  often  than  not,  the  product  was  too  old  to  use  or  simply  unavailable.  As  far  as  the 
accuracy  of  the  icing  potential,  the  product  I  had  the  most  amount  of  time  to  use,  it  did  not 
verify  very  well  in  the  Western  US.  Needs  fewer  layers  at  this  point. _ 
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Question  -  List  other  products  that  you  think  should  be  added  to  IIDA. 


Icing  intensity 

The  products  available  are  adequate. 

this  questionaire  will  only  let  you  comment  with  a  limited  number  of  characters,  even  though 
there  is  still  plenty  of  room  left  in  the  text  box! 

stability/vertical  motion 

Air  Force  Iceing  analysis  on  realtime  RAOBS. 

I’d  like  to  see  the  IIDA  run  on  some  other  models  which  have  better  RH  forecasts  to  see  if  it  has 
more  utility. 

need  an  algorithm  to  project  and  forecast  icg  for  the  next  6-8  hrs  or  so. 

Question  -  List  suggestions  for  improving  any  of  the  components.  Please  be  sure  to 
include  the  component  name  in  the  suggestion(s). 

icg  tops  in  the  centeral  ptn  of  the  US  are  consistantly  too  high  by  4-6,000  ft.  icg  tops  and  cloud 
tops  could  be  the  two  most  useful  outputs  from  the  iida,  but  the  accuracy  has  been  poor.  SLD 
hasn’t  seemed  to  relate  to  ice  type  or  intensity. 

Remove  ice  reports  from  the  data  base  near  thunderstorms.  Determine  type  of  aircraft  and 
use  to  define  intensity  by  aircraft  type. 

Have  forecast  fields  out  to  1 2  hours  for  all  of  the  components 

It’s  probably  my  fault,  but  1  did  not  know  what  "Icing  Cloud  Hgts  and  Bases  "  means. 

Already  listed  several  suggestions  in  previous  answers 

Add  a  forecast  capability,  possibly  based  on  TAFs  or  model  data. 

Base  and  top  thresholds  too  low. 

Do  not  see  need  for  1 0  percent  intervals--makes  appearance  too  blotchy. 

A  composite  layer  product  for  icing  potential  with  the  ability  to  look  at  smaller  layers  when 
necessary.  Or  a  symbol  at  certain  grid  points  which  has  the  top  and  base  depicted,  and  the 
ability  to  overlay  this  on  top  of  the  icing  potential/other  product. 
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IIDA  ADDED  VALUE  IDENTIFIED  FROM  THE 
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Question  -  Does  the  integrated  product  provide  any  value  beyond  having  the  individual 
components  by  themselves?  Please  explain.  Provide  examples  if  necessary. _ 

in  theory,  it  should  be  best  to  integrate  the  base/tops/cloud  and  model  info  into  one  product  to 

provide  a  one-stop  look  at  the  icing  layer,  the  accuracy  has  been  disappointing,  though. _ 

It  is  similar  to  the  NCAR  RAP  and  neural  network  icing  in  depicting  a  threat  area. _ 

In  the  winter  there  is  often  little  time  to  devote  to  the  Icing  forecasts.  Graphics  that  summarize 

or  otherwise  depict  data  from  several  fields  are  appreciated. _ _ 

Not  much  additional  information  here  than  from  other  products. _ 

Reduces  number  of  seperate  screens  to  look  at. _ 

Integrated  product  is  better  than  the  individual  components.  The  individual  components  could 

be  better  which  would  improve  the  integrated  product. _ 

can  save  time  in  determining  icg  airmet.  but  this  also  may  reduce  the  accuracy  somewhat. 
Because  of  time  factor,  we  are  often  unable  to  look  at  so  many  components  in  preparing 

airmets/sigmets. _ 

A  lot  faster  to  use. 


Question  -  Does  the  IIDA  provide  any  value  above  what  you  currently  use  to  create  Icing 
AIRMETs  (e.g.,  satellite,  raar,  model  soundings,  PIREPs)?  How? _ 

it  basically  draws  the  same  large  area  that  we  get  from  other  “cold  cloud"  forecasting  tools 

icemap  and  other  current  products  are  as  good  or  better. _ 

It  does  not  add  any  information  to  tools  and  algorithms  already  available  to  the  AWC _ 

It  helps  some,  but  not  a  great  deal. _ 

I  do  not  much  additional  value  to  what  other  products  are  currently  available. _ 

It  has  no  forecast  function. _ 

AWC  icing  tools  are  more  advanced  than  IIDA. _ 

To  be  honest,  I  found  little  value  over  the  tools  we  already  have.  When  forecast  tools  are 

available  from  other  data  sources,  these  prove  to  be  much  more  useful  in  my  view. _ 

It  seems  to  reflect  what  we  can  gather  from  other  sources.. .ie  it’s  nothing  new  and  improved  or 
above  and  beyond  the  rest  of  the  tools  we  use,  though  I  did  like  the  graphics  display. _ 


Question  -  Does  the  IIDA  reduce  the  amount  of  time  necessary  to  create  an  Icing 
A1RMET?  How? _ 

no,  because  it  is  not  related  to  intensity _ 

pireps  during  the  day  are  the  biggest  help. _ 

It  does  not  add  any  information  to  tools  and  algorithms  already  available  to  the  AWC _ 

Since  it  is  only  a  diagostice,  it  lenthens  the  process  of  formulating  and  AIRMET.  It  had  too,  it  is 

another  tool  to  look  at.  It  didnot  take  the  place  of  anything  else. _ _ 

Product  not  timely  many  times  and  out  of  date  when  producing  airmets.  Greater  timeliness 

would  be  first  large  step  in  a  producing  an  additional  in-depth  evaluation  in  the  future. _ 

It  provides  a  minimal  amount  of  information,  but  takes  too  much  time  to  use. _ 

I  still  need  to  look  at  other  tools, _ 

it  does  reduce  the  time..but  i  do  not  have  complete  confidence  to  rely  on  it  totally,  i  still  prefer  to 
look  at  satellite,  radar,  model  soundings  and  pireps  and  also  the  previous  history  of  where  the 

icq  has  been  ovr  the  past  several  hours. _ 

As  with  any  new  product/technology,  a  longer  period  of  evaluation  or  actual  use  would  be 

necessary  to  determine  if  time  could  be  saved  using  this  product. _ 

icing  rules! _ 

If  the  product  displayed  the  data  with  information  overlays,  it  could  be  faster,  but  with  the 
misses  it  has  on  accuracy,  one  would  spend  more  time  checking  to  see  if  it  is  believable.  It  has 
potential,  (no  pun  intended)  but  not  quite  there  yet _ 


